Back

Generalized Out-of-Distribution Detection: A Survey

Jingkang YangKaiyang ZhouYixuan LiZiwei Liu

article2021IJCV1,542 citations

Establishes a unified taxonomy that connects out-of-distribution detection with anomaly detection, novelty detection, open set recognition, and outlier detection to clarify problem boundaries and guide future methodology.

Listen

Modern machine learning models are typically trained under the closed-world assumption that operational data will match training data. In open-world deployments, however, models frequently encounter out-of-distribution inputs that can cause dangerous, overconfident misclassifications. This issue poses serious safety and operational risks in critical systems such as autonomous driving and automated diagnostics. Although related sub-fields—including anomaly detection, novelty detection, open set recognition, out-of-distribution detection, and outlier detection—address this core challenge, they have historically evolved in isolation with fragmented terminology and inconsistent benchmarks.

The article establishes a unified framework, termed generalized out-of-distribution detection, to systematically categorize and compare these five distinct sub-fields. It specifically aims to clarify their theoretical differences, survey the rapid methodological developments in out-of-distribution detection, and benchmark representative techniques under standardized experimental conditions.

The authors analyze the problem space using four clear criteria: whether the distribution shift is sensory (covariate) or semantic (label-based), whether the target distribution contains single or multiple classes, whether in-distribution classification must be preserved, and whether learning is inductive (train-then-test) or transductive (evaluating all data together). Using this taxonomy, the article conducts a comprehensive literature review across classification-based, density-based, distance-based, and reconstruction-based methods. To support fair comparison, the authors evaluate these methods on standardized vision benchmarks using the OpenOOD platform with unified backbones and training configurations.

The analysis yields four central findings. First, simple model uncertainty and data augmentation strategies during training, such as PixMix and CutMix, prove exceptionally effective, with PixMix achieving a leading 93.1% AUROC score on challenging near-distribution benchmarks. Second, inference-only, post-hoc methods—such as deep nearest neighbors (KNN) and activation rectifications—consistently match or outperform specialized training-heavy approaches while avoiding retraining overhead. Third, exposing models to auxiliary outlier datasets during training provides marginal practical advantage over modern outlier-free, post-hoc methods, while introducing severe risks of training contamination. Finally, specialized anomaly detection methods designed for pixel-level irregularities transfer effectively to identifying far out-of-distribution samples.

These findings have direct operational and economic implications for deploying reliable artificial intelligence systems. Organizations do not necessarily need to invest heavy compute budgets into retraining models with massive synthetic or auxiliary datasets to achieve safe decision-making. Instead, deploying modular, post-hoc inference filters over well-regularized base classifiers offers a cost-effective path to significantly mitigate the risk of high-confidence failures on novel data.

Decision-makers should prioritize integrating non-parametric, post-hoc detection algorithms alongside standard data-augmentation pipelines when deploying safety-critical vision systems. Future technical efforts should move away from relying on uncurated auxiliary outlier sets and focus on full-spectrum detection, which ensures systems simultaneously generalize to harmless sensory variations while reliably rejecting true semantic novelties. Further validation should be conducted on large-scale benchmarks like ImageNet and real-world multi-modal tasks.

Confidence in these findings is high for standard computer vision classification benchmarks, supported by rigorous, standardized evaluations in OpenOOD. However, stakeholders should remain cautious when translating these conclusions directly to large language models, dense perception tasks like segmentation, or domain-specific settings where benchmark contamination and fine-grained semantic overlaps remain active research challenges.

Nanyang Technological UniversityUniversity of Wisconsin Madisonanomaly detectionautonomous drivingclassificationgeneralized out-of-distribution detectionanomaly detectionmulti-class novelty detection
arXiv: 2110.11334Jingkang50/OODSurvey

No sufficiently relevant recommendations were found.

Cover for Generalized Out-of-Distribution Detection: A Survey

Abstract

Out-of-distribution (OOD) detection is critical to ensuring the reliability and safety of machine learning systems. For instance, in autonomous driving, we would like the driving system to issue an alert and hand over the control to humans when it detects unusual scenes or objects that it has never seen during training time and cannot make a safe decision. The term, OOD detection, first emerged in 2017 and since then has received increasing attention from the research community, leading to a plethora of methods developed, ranging from classification-based to density-based to distance-based ones. Meanwhile, several other problems, including anomaly detection (AD), novelty detection (ND), open set recognition (OSR), and outlier detection (OD), are closely related to OOD detection in terms of motivation and methodology. Despite common goals, these topics develop in isolation, and their subtle differences in definition and problem setting often confuse readers and practitioners. In this survey, we first present a unified framework called generalized OOD detection, which encompasses the five aforementioned problems, i.e., AD, ND, OSR, OOD detection, and OD. Under our framework, these five problems can be seen as special cases or sub-tasks, and are easier to distinguish. We then review each of these five areas by summarizing their recent technical developments, with a special focus on OOD detection methodologies. We conclude this survey with open challenges and potential research directions.

Topics

anomaly detectionautonomous drivingclassificationdetection algorithmout-of-distribution detectionranging

Keywords

generalized out-of-distribution detectionanomaly detectionmulti-class novelty detectionopen-world recognitionopen category detectioncovariate shift detectionoutlier exposureoutlier synthesispost-hoc detection

Table of Contents

  • 1 Introduction
  • 2 Generalized OOD Detection
  • 2.1 Anomaly Detection
  • 2.2 Novelty Detection
  • 2.3 Open Set Recognition
  • 2.4 Out-of-Distribution Detection
  • 2.5 Outlier Detection
  • 2.6 Related Topics
  • 2.7 Organization of Remaining Sections
  • 3 OOD Detection: Methodology
  • 3.1 Classification-based Methods
  • 3.1.1 Output-based Methods
  • 3.1.2 Methods with Outlier Exposure
  • 3.1.3 Gradient-based Methods
  • 3.1.4 Bayesian Models
  • 3.1.5 OOD Detection for Foundation Models
  • 3.2 Density-based Methods
  • 3.3 Distance-based Methods
  • 3.4 Reconstruction-based Methods
  • 3.5 Theoretical Analysis
  • 3.6 Discussion
  • 4 Methodologies from Other Sub-tasks
  • 4.1 Open Set Recognition
  • 4.2 Anomaly Detection & Novelty Detection
  • 4.3 Outlier Detection
  • 5 Benchmarks and Experiments
  • 5.1 Benchmarks and Metrics
  • 5.2 Experimental Setup
  • 5.3 Experimental Results and Findings
  • 5.4 Exclusion of Covariate-Shift Detection
  • 6 Challenges and Future Directions
  • 6.1 Challenges
  • 6.2 Future Directions
  • 7 Conclusion
  • References

Knowls

  1. Knowl 1 — Taxonomy of Generalized Out-of-Distribution Detection

    definition

    Generalized out-of-distribution (OOD) detection is a unified taxonomy that categorizes open-world detection problems into five sub-tasks: Anomaly Detection (AD), Novelty Detection (ND), Open Set Recognition (OSR), Out-of-Distribution (OOD) Detection, and Outlier Detection (OD). The classification of these tasks is determined by four foundational dichotomies:

    1. Type of Distribution Shift:

      • Covariate shift: Changes occur only in the input distribution P(X)P(X) while the label distribution P(Y)P(Y) remains unchanged (e.g., domain shifts, image corruptions).
      • Semantic shift: Changes occur in the label space P(Y)P(Y) (e.g., the emergence of novel categories), which naturally induces a shift in P(X)P(X).

    2. In-Distribution (ID) Data Cardinality:

      • Single-class: ID data represents a single homogeneous class (common in classic AD and one-class ND).
      • Multi-class: ID data contains multiple distinct semantic categories (standard in OSR, multi-class ND, and OOD detection).

    3. ID Classification Requirement:

      • Required: The system must simultaneously maintain high closed-set multi-class classification accuracy on ID samples while detecting OOD instances (as in OSR and OOD detection).
      • Not required: The task is formulated purely as binary identification (ID vs. non-ID), without requiring differentiation among individual ID classes (as in AD and ND).

    4. Learning Paradigm:

      • Inductive: Follows a train-test split where the ID distribution is learned during training, and unseen samples are evaluated during inference (AD, ND, OSR, OOD detection).
      • Transductive: All observations (contaminated with an unknown fraction of anomalies) are provided simultaneously without a separate training set, and the goal is to identify the minority anomalies from the majority ID data (as in OD).

  2. Knowl 2 — Formal Formulations of Generalized OOD Detection Sub-tasks

    definition

    Let X\mathcal{X} denote the input space of observations and Y\mathcal{Y} denote the label space of semantic classes. Training data is drawn from an in-distribution (ID) joint probability distribution P(X,Y)P(X, Y), and test data is drawn from a potentially shifted distribution P′(X,Y)P'(X, Y):

    • Sensory Anomaly Detection (Sensory AD): Focuses exclusively on covariate shift on low-level sensory observations without semantic label changes: P′(X)≠P(X)andP′(Y)=P(Y)P'(X) \neq P(X) \quad \text{and} \quad P'(Y) = P(Y) The ID training set is treated as a single class, and ID classification is not required.

    • Semantic Anomaly Detection / Novelty Detection (ND): Focuses on detecting samples from novel semantic categories: P′(Y)≠P(Y)P'(Y) \neq P(Y) ND does not require classifying the ID classes, even if multiple classes are present in the training data.

    • Open Set Recognition (OSR): Given an ID training set with multiple classes Y={1,…,C}\mathcal{Y} = \{1, \dots, C\}, the model must simultaneously perform accurate CC-way closed-set classification for ID inputs (Y∈YY \in \mathcal{Y}) and reject test inputs exhibiting semantic shift (Y∉YY \notin \mathcal{Y}).

    • Out-of-Distribution (OOD) Detection: Formulated canonically identically to OSR for multi-class classification (P′(Y)≠P(Y)P'(Y) \neq P(Y) while preserving ID classification performance), but generalized to broader vision and learning tasks (such as multi-label classification, object detection, and segmentation) and evaluated across multi-dataset distribution shifts.

    • Outlier Detection (OD): Operates transductively on an unpartitioned dataset D={xi}i=1N\mathcal{D} = \{\mathbf{x}_i\}_{i=1}^N to identify anomalous samples that deviate significantly from the majority distribution, driven by either covariate or semantic shift.

  3. Knowl 3 — Distinctions Between Open Set Recognition and OOD Detection

    definition

    While Open Set Recognition (OSR) and Out-of-Distribution (OOD) detection both address semantic shift detection while maintaining in-distribution (ID) classification, they differ in three key aspects:

    1. Evaluation Protocol and Benchmark Design: OSR traditionally constructs benchmarks by partitioning a single multi-class dataset into known (ID) and unknown (OOD) classes based on label splits. In contrast, OOD detection designates an entire dataset (e.g., CIFAR-10) as ID and employs entirely separate datasets (e.g., SVHN, Textures) as OOD, with strict de-duplication to eliminate overlapping classes.

    2. Outlier Data Constraints During Training: By theoretical formulation (such as bounding open-space risk), OSR restricts model training strictly to ID data and discourages the use of external outlier data during training. OOD detection accommodates a wider algorithmic space, including methods utilizing outlier exposure (OE) from real auxiliary datasets or synthesized negative data.

    3. Scope and Solution Space: OSR is predominantly formulated around closed-set classifiers calibrated for reject options, whereas OOD detection extends across multi-label classification, dense prediction tasks (e.g., object detection, segmentation), and multi-modal foundation models.

  4. Knowl 4 — Post-Hoc Output and Activation Manipulation for OOD Detection

    model/method

    Post-hoc OOD detection methods operate on pre-trained neural networks without modifying the training loss or architecture:

    • Maximum Softmax Probability (MSP): Uses the maximum softmax probability as the confidence score, identifying inputs with low maximum probability as OOD.

    • ODIN: Applies temperature scaling T>1T > 1 and input gradient perturbation to widen the softmax score divergence between ID and OOD samples: Si(x;T)=exp⁡(fi(x)/T)∑j=1Cexp⁡(fj(x)/T)S_i(\mathbf{x}; T) = \frac{\exp(f_i(\mathbf{x}) / T)}{\sum_{j=1}^C \exp(f_j(\mathbf{x}) / T)} where fi(x)f_i(\mathbf{x}) is the logit for class ii.

    • Energy-Based OOD Detection: Maps the logits to a scalar Helmholtz free energy score, theoretically aligned with the input's negative log marginal likelihood −log⁡p(x)- \log p(\mathbf{x}): E(x;f)=−T⋅log⁡∑i=1Cexp⁡(fi(x)T)E(\mathbf{x}; f) = -T \cdot \log \sum_{i=1}^C \exp\left(\frac{f_i(\mathbf{x})}{T}\right) Inputs with higher energy scores E(x;f)E(\mathbf{x}; f) are flagged as OOD.

    • Rectified Activations (ReAct): Identifies that mismatched batch normalization statistics from OOD inputs trigger abnormally high activations in penultimate feature layers. ReAct clips unit activations at a threshold cc before the classification layer: hk∗(x)=min⁡(hk(x),c)h_k^*(x) = \min(h_k(x), c) where hk(x)h_k(x) is the activation of the kk-th feature dimension.

    • Directed Sparsification (DICE): Prunes noisy, non-salient network weights based on contribution rankings, reducing output variance on OOD data and sharpening the ID/OOD separation.

    • Activation Shaping (ASH): Removes a large fraction (e.g., 90%90\%) of late-layer feature representations based on a top-KK criterion and adjusts the remaining activations by scaling or constant assignments.

  5. Knowl 5 — Distance-Based OOD Scoring: Mahalanobis and Deep Nearest Neighbors

    model/method

    Distance-based OOD detection measures the proximity of test features to in-distribution representations in the intermediate or penultimate feature space of a neural network:

    • Parametric Mahalanobis Distance: Assumes that pre-trained deep feature representations z=f(x)\mathbf{z} = f(\mathbf{x}) follow a class-conditional Gaussian distribution N(μc,Σ)\mathcal{N}(\boldsymbol{\mu}_c, \boldsymbol{\Sigma}) with shared covariance Σ\boldsymbol{\Sigma} across classes c∈{1,…,C}c \in \{1, \dots, C\}. The confidence score is defined by the minimum Mahalanobis distance to any class centroid: M(x)=max⁡c−(z−μc)⊤Σ−1(z−μc)M(\mathbf{x}) = \max_{c} -(\mathbf{z} - \boldsymbol{\mu}_c)^\top \boldsymbol{\Sigma}^{-1} (\mathbf{z} - \boldsymbol{\mu}_c)

    • Non-Parametric Nearest-Neighbor Distance (KNN): Computes the Euclidean distance between the normalized penultimate feature representation z∗=z/∥z∥2\mathbf{z}^* = \mathbf{z} / \|\mathbf{z}\|_2 of a test sample x\mathbf{x} and the kk-th nearest neighbor in the stored set of normalized training embeddings {zi∗}i=1N\{\mathbf{z}_i^*\}_{i=1}^N: Dk(x)=∥z∗−z(k)∗∥2D_k(\mathbf{x}) = \|\mathbf{z}^* - \mathbf{z}_{(k)}^*\|_2 Non-parametric distance scoring avoids parametric Gaussian assumptions, making it robust across high-dimensional feature spaces and complex multi-modal class distributions.

  6. Knowl 6 — Feature-Space Virtual Outlier Synthesis for Outlier Exposure

    model/method

    While outlier exposure (OE) using auxiliary real datasets significantly improves ID/OOD separability, collecting clean, non-overlapping real outliers is often impractical, and pixel-space GAN synthesis is computationally expensive in high dimensions.

    Feature-space synthesis models the feature representations of in-distribution data directly in the latent space Z\mathcal{Z}:

    • Parametric Virtual Outlier Synthesis (VOS): Estimates class-conditional Gaussian distributions N(μc,Σc)\mathcal{N}(\boldsymbol{\mu}_c, \boldsymbol{\Sigma}_c) over penultimate layer features and samples virtual outliers from the low-likelihood boundary region of the feature space. An auxiliary energy-regularization loss forces the classifier to produce high energy on these synthesized virtual representations during training.

    • Non-Parametric Outlier Synthesis (NPOS): Generates synthetic outliers in feature space along the boundary of non-parametric ID nearest-neighbor envelopes without assuming Gaussianity.

  7. Knowl 7 — Near-OOD vs. Far-OOD Benchmarking Protocol with Semantic De-duplication

    experimental setup

    In the standard OpenOOD benchmarking evaluation, OOD detection datasets are partitioned into Near-OOD and Far-OOD sets relative to an In-Distribution (ID) dataset:

    • Near-OOD: Test sets that share the same general visual domain as the ID dataset but differ purely by semantic categories (e.g., CIFAR-100 and TinyImageNet when CIFAR-10 is used as ID).
    • Far-OOD: Test sets that exhibit both semantic shift and severe covariate/domain shifts (e.g., MNIST, SVHN, Textures, and Places365 when CIFAR-10 is used as ID).

    Semantic De-duplication Requirement: To prevent corrupted evaluation metrics (where true ID samples in the OOD set are falsely penalized when classified with high confidence), overlapping classes must be explicitly purged from OOD datasets:

    • When evaluating on CIFAR-10, 1,207 semantically overlapping images are removed from TinyImageNet, and 1,305 overlapping images are removed from Places365.
    • When evaluating on CIFAR-100, 2,502 semantically overlapping images are removed from TinyImageNet, and 1,305 overlapping images are removed from Places365.

  8. Knowl 8 — Empirical Performance Comparison Across Generalized OOD Detection Families

    empirical result

    Standardized evaluation on ResNet-18 backbones trained on CIFAR-10 and CIFAR-100 across various method categories reveals several empirical findings on the OpenOOD benchmark:

    1. Data Augmentation and Uncertainty Techniques: Methods utilizing rich data augmentation—specifically PixMix, CutMix, and Mixup—demonstrate superior performance. PixMix attains the top AUROC score of 93.1%93.1\% on CIFAR-10 Near-OOD.

    2. Post-Hoc Methods vs. Training-Required Methods: Modern inference-only post-hoc methods (such as KNN, ReAct, and DICE) consistently match or exceed the detection performance of specialized training-required objectives while requiring no retraining overhead.

    3. Utility of Auxiliary Outlier Data: Non-parametric methods without outlier data (e.g., KNN) achieve performance competitive with or superior to outlier exposure methods (e.g., UDG) on Far-OOD and CIFAR-100 Near-OOD, indicating that large auxiliary datasets are not strictly necessary for strong general OOD detection.

    4. Sensory Anomaly Detection Methods on Far-OOD: Anomaly detection techniques engineered for pixel-level defect identification (such as DRAEM and CutPaste) exhibit strong transferability to Far-OOD detection tasks on CIFAR-100 benchmarks.

  9. Knowl 9 — Covariate Shift Sensitivity Dilemma and Full-Spectrum OOD Detection

    limitation

    A fundamental limitation of existing OOD detectors is their unintended sensitivity to benign covariate shifts (e.g., image corruptions, style changes, domain shifts) rather than purely semantic shifts. Because deep models often rely on non-semantic environmental features, standard scoring functions frequently trigger false OOD alarms for covariate-shifted in-distribution samples that the model could otherwise correctly classify.

    To address this dilemma, the full-spectrum OOD detection framework requires the classifier to be robust and generalize correctly across covariate-shifted ID inputs (P′(X)≠P(X),P′(Y)=P(Y)P'(X) \neq P(X), P'(Y) = P(Y)) while reliably rejecting samples that contain true semantic shifts (P′(Y)≠P(Y)P'(Y) \neq P(Y)).

Coverage note — Omitted detailed descriptions of general background surveys on non-deep tabular outlier detection heuristics (e.g., LOF, RANSAC, Isolation Forest) and specific hyperparameter tables from cited external papers, as they represent well-established prior literature rather than core contributions of the generalized framework.

References

  1. 1.D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Man´e, “Concrete problems in AI safety,” arXiv preprint arXiv:1606.06565, 2016.
  2. 2.S. Mohseni, H. Wang, Z. Yu, C. Xiao, Z. Wang, and J. Yadawa, “Practical machine learning safety: A survey and primer,” arXiv preprint arXiv:2106.04823, 2021.
  3. 3.D. Hendrycks, N. Carlini, J. Schulman, and J. Steinhardt, “Unsolved problems in ML safety,” arXiv preprint arXiv:2109.13916, 2021.
  4. 4.D. Hendrycks and M. Mazeika, “X-risk analysis for AI research,” arXiv preprint arXiv:2206.05862, 2022.
  5. 5.A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in NIPS, 2012.
  6. 6.K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in ICCV, 2015.
  7. 7.N. Drummond and R. Shearer, “The open world assumption,” in eSI Workshop, 2006.
  8. 8.D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” in ICLR, 2017.
  9. 9.S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan, “A theory of learning from different domains,” Machine learning, 2010.
  10. 10.D. Li, Y. Yang, Y.-Z. Song, and T. M. Hospedales, “Deeper, broader and artier domain generalization,” in ICCV, 2017.
  11. 11.M. Wang and W. Deng, “Deep visual domain adaptation: A survey,” Neurocomputing, 2018.
  12. 12.C. C. Aggarwal and P. S. Yu, “Outlier detection for high dimensional data,” in ACM SIGMOD, 2001.
  13. 13.V. Hodge and J. Austin, “A survey of outlier detection methodologies,” Artificial intelligence review, 2004.
  14. 14.I. Ben-Gal, “Outlier detection,” in Data mining and knowledge discovery handbook, 2005.
  15. 15.H. Wang, M. J. Bah, and M. Hammad, “Progress in outlier detection techniques: A survey,” Ieee Access, 2019.
  16. 16.L. Ruff, J. R. Kauffmann, R. A. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T. G. Dietterich, and K.-R. M¨uller, “A unifying review of deep and shallow anomaly detection,” Proceedings of the IEEE, 2021.
  17. 17.G. Pang, C. Shen, L. Cao, and A. v. d. Hengel, “Deep learning for anomaly detection: A review,” arXiv preprint arXiv:2007.02500, 2020.
  18. 18.S. Bulusu, B. Kailkhura, B. Li, P. K. Varshney, and D. Song, “Anomalous example detection in deep learning: A survey,” IEEE Access, 2020.
  19. 19.R. Chalapathy and S. Chawla, “Deep learning for anomaly detection: A survey,” arXiv preprint arXiv:1901.03407, 2019.
  20. 20.M. A. Pimentel, D. A. Clifton, L. Clifton, and L. Tarassenko, “A review of novelty detection,” Signal Processing, 2014.
  21. 21.D. Miljkovi´c, “Review of novelty detection methods,” in MIPRO, 2010.
  22. 22.M. Markou and S. Singh, “Novelty detection: a review—part 1: statistical approaches,” Signal processing, 2003.
  23. 23.M. Markou and S. Singh, “Novelty detection: a review—part 2: neural network based approaches,” Signal processing, 2003.
  24. 24.T. E. Boult, S. Cruz, A. R. Dhamija, M. Gunther, J. Henrydoss, and W. J. Scheirer, “Learning and the unknown: Surveying steps toward open world recognition,” in AAAI, 2019.
  25. 25.C. Geng, S.-j. Huang, and S. Chen, “Recent advances in open set recognition: A survey,” TPAMI, 2020.
  26. 26.A. Mahdavi and M. Carvalho, “A survey on open set recognition,” arXiv preprint arXiv:2109.00893, 2021.
  27. 27.I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in ICLR, 2015.
  28. 28.A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” ICLR, 2018.
  29. 29.J. Qui˜nonero-Candela, M. Sugiyama, N. D. Lawrence, and A. Schwaighofer, Dataset shift in machine learning. Mit Press, 2009.
  30. 30.L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in CVPR, 2016.
  31. 31.V. Chandola, A. Banerjee, and V. Kumar, “Anomaly detection: A survey,” ACM computing surveys (CSUR), vol. 41, no. 3, pp. 1–58, 2009.
  32. 32.F. Ahmed and A. Courville, “Detecting semantic anomalies,” in AAAI, 2020.
  33. 33.L. Zhang, M. Goldstein, and R. Ranganath, “Understanding failures in out-of-distribution detection with deep generative models,” in ICML, 2021.
  34. 34.N. Akhtar and A. Mian, “Threat of adversarial attacks on deep learning in computer vision: A survey,” IEEE Access, 2018.
  35. 35.K. Patel, H. Han, and A. K. Jain, “Secure face unlock: Spoof detection on smartphones,” IEEE transactions on information forensics and security, 2016.
  36. 36.D. Wen, H. Han, and A. K. Jain, “Face spoof detection with image distortion analysis,” IEEE Transactions on Information Forensics and Security, 2015.
  37. 37.K. A. Nixon, V. Aimale, and R. K. Rowe, “Spoof detection schemes,” in Handbook of biometrics, 2008.
  38. 38.G. Polatkan, S. Jafarpour, A. Brasoveanu, S. Hughes, and I. Daubechies, “Detection of forgery in paintings using supervised learning,” in ICIP, 2009.
  39. 39.B. Dolhansky, R. Howes, B. Pflaum, N. Baram, and C. C. Ferrer, “The deepfake detection challenge (dfdc) preview dataset,” arXiv preprint arXiv:1910.08854, 2019.
  40. 40.L. Jiang, Z. Guo, W. Wu, Z. Liu, Z. Liu, C. C. Loy, S. Yang, Y. Xiong, W. Xia, B. Chen, P. Zhuang, S. Li, S. Chen, T. Yao, S. Ding, J. Li, F. Huang, L. Cao, R. Ji, C. Lu, and G. Tan, “DeeperForensics Challenge 2020 on real-world face forgery detection: Methods and results,” arXiv preprint arXiv:2102.09471, 2021.
  41. 41.P. Yang, D. Baracchi, R. Ni, Y. Zhao, F. Argenti, and A. Piva, “A survey of deep learning-based source image forensics,” Journal of Imaging, 2020.
  42. 42.P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger, “Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection,” in CVPR, 2019.
  43. 43.W.-H. Chu and K. M. Kitani, “Neural batch sampling with reinforcement learning for semi-supervised anomaly detection,” in ECCV, 2020.
  44. 44.D. J. Atha and M. R. Jahanshahi, “Evaluation of deep learning approaches based on convolutional neural networks for corrosion detection,” Structural Health Monitoring, 2018.
  45. 45.H. Idrees, M. Shah, and R. Surette, “Enhancing camera surveillance using computer vision: a research note,” Policing: An International Journal, 2018.
  46. 46.C. P. Diehl and J. B. Hampshire, “Real-time object classification and novelty detection for collaborative video surveillance,” in IJCNN, 2002.
  47. 47.L.-J. Li and L. Fei-Fei, “Optimol: automatic online picture collection via incremental model learning,” IJCV, 2010.
  48. 48.D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” in ICLR, 2017.
  49. 49.T. Fawcett, “An introduction to roc analysis,” Pattern recognition letters, 2006.
  50. 50.D. M. Powers, “Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation,” JMLT, 2020.
  51. 51.H. R. Kerner, D. F. Wellington, K. L. Wagstaff, J. F. Bell, C. Kwan, and H. B. Amor, “Novelty detection for multispectral images with application to planetary exploration,” in AAAI, 2019.
  52. 52.H. Al-Behadili, A. Grumpe, and C. W¨ohler, “Incremental learning and novelty detection of gestures in a multi-class system,” in AIMS, 2015.
  53. 53.D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, “Curiosity-driven exploration by self-supervised prediction,” in ICML, 2017.
  54. 54.P. Perera, R. Nallapati, and B. Xiang, “Ocgan: One-class novelty detection using gans with constrained latent representations,” in CVPR, 2019.
  55. 55.Y. Xia, X. Cao, F. Wen, G. Hua, and J. Sun, “Learning discriminative reconstructions for unsupervised outlier removal,” in CVPR, 2015.
  56. 56.W. J. Scheirer, A. de Rezende Rocha, A. Sapkota, and T. E. Boult, “Toward open set recognition,” TPAMI, 2013.
  57. 57.A. R. Dhamija, M. G¨unther, and T. E. Boult, “Reducing network agnostophobia,” in NeurIPS, 2018.
  58. 58.S. Liu, R. Garrepalli, T. Dietterich, A. Fern, and D. Hendrycks, “Open category detection with pac guarantees,” in ICML, 2018.
  59. 59.Z. Fang, J. Lu, A. Liu, F. Liu, and G. Zhang, “Learning bounds for open-set learning,” in ICML, 2021.
  60. 60.E. Sorio, A. Bartoli, G. Davanzo, and E. Medvet, “Open world classification of printed invoices,” in Proceedings of the 10th ACM symposium on Document engineering, 2010.
  61. 61.H. Xu, B. Liu, L. Shu, and P. Yu, “Open-world learning and application to product classification,” in WWW, 2019.
  62. 62.X. Huang, D. Kroening, W. Ruan, J. Sharp, Y. Sun, E. Thamo, M. Wu, and X. Yi, “A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability,” Computer Science Review, 2020.
  63. 63.A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in CVPR, 2012.
  64. 64.D. Hendrycks, M. Mazeika, and T. Dietterich, “Deep anomaly detection with outlier exposure,” in ICLR, 2019.
  65. 65.J. Zhang, N. Inkawhich, R. Linderman, Y. Chen, and H. Li, “Mixture outlier exposure: Towards out-of-distribution detection in fine-grained environments,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 5531–5540, January 2023.
  66. 66.D. Hendrycks, S. Basart, M. Mazeika, M. Mostajabi, J. Steinhardt, and D. Song, “Scaling out-of-distribution detection for real-world settings,” in ICML, 2022.
  67. 67.Y.-C. Hsu, Y. Shen, H. Jin, and Z. Kira, “Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data,” in CVPR, 2020.
  68. 68.G. Pleiss, A. Souza, J. Kim, B. Li, and K. Q. Weinberger, “Neural network out-of-distribution detection for regression tasks,” 2019.
  69. 69.O. Vinyals, T. Ewalds, S. Bartunov, P. Georgiev, A. S. Vezhnevets, M. Yeo, A. Makhzani, H. K¨uttler, J. Agapiou, J. Schrittwieser, et al., “Starcraft ii: A new challenge for reinforcement learning,” arXiv preprint arXiv:1708.04782, 2017.
  70. 70.A. Sedlmeier, T. Gabor, T. Phan, L. Belzner, and C. Linnhoff-Popien, “Uncertainty-based out-of-distribution detection in deep reinforcement learning,” arXiv preprint arXiv:1901.02219, 2019.
  71. 71.D. Zimmerer, P. M. Full, F. Isensee, P. J¨ager, T. Adler, J. Petersen, G. K¨ohler, T. Ross, A. Reinke, A. Kascenas, et al., “Mood 2020: A public benchmark for out-of-distribution detection and localization on medical images,” IEEE Transactions on Medical Imaging, 2022.
  72. 72.M. I. Tariq, N. A. Memon, S. Ahmed, S. Tayyaba, M. T. Mushtaq, N. A. Mian, M. Imran, and M. W. Ashraf, “A review of deep learning security and privacy defensive techniques,” Mobile Information Systems, vol. 2020, 2020.
  73. 73.R. Averly and W.-L. Chao, “Unified out-of-distribution detection: A model-specific perspective,” arXiv preprint arXiv:2304.06813, 2023.
  74. 74.Wikipedia contributors, “Outlier from Wikipedia, the free encyclopedia,” 2021. [Online; accessed 12 August 2021].
  75. 75.M. Bianchini, A. Belahcen, and F. Scarselli, “A comparative study of inductive and transductive learning with feedforward neural networks,” in Conference of the Italian Association for Artificial Intelligence, 2016.
  76. 76.I. Ben-Gal, “Outlier detection,” in Data mining and knowledge discovery handbook, 2005.
  77. 77.S. Basu and M. Meckesheimer, “Automatic outlier detection for time series: an application to sensor data,” Knowledge and Information Systems, 2007.
  78. 78.Y. Dou, W. Li, Z. Liu, Z. Dong, J. Luo, and S. Y. Philip, “Uncovering download fraud activities in mobile app markets,” in ASONAM, 2019.
  79. 79.T. Xiao, C. Zhang, and H. Zha, “Learning to detect anomalies in surveillance video,” IEEE Signal Processing Letters, 2015.
  80. 80.H. Liu, S. Shah, and W. Jiang, “On-line outlier detection and data cleaning,” Computers & chemical engineering, 2004.
  81. 81.A. Loureiro, L. Torgo, and C. Soares, “Outlier detection using clustering methods: a data cleaning application,” in Proceedings of KDNet Symposium on Knowledge-based Systems, 2004.
  82. 82.J. Van den Broeck, S. Argeseanu Cunningham, R. Eeckels, and K. Herbst, “Data cleaning: detecting, diagnosing, and editing data abnormalities,” PLoS medicine, 2005.
  83. 83.Y. Wang, W. Liu, X. Ma, J. Bailey, H. Zha, L. Song, and S.-T. Xia, “Iterative learning with open-set noisy labels,” in CVPR, 2018.
  84. 84.X. Chen and A. Gupta, “Webly supervised learning of convolutional networks,” in ICCV, 2015.
  85. 85.K. Cao, M. Brbic, and J. Leskovec, “Open-world semi-supervised learning,” arXiv preprint arXiv:2102.03526, 2021.
  86. 86.P. L. Bartlett and M. H. Wegkamp, “Classification with a reject option using a hinge loss.,” Journal of Machine Learning Research, vol. 9, no. 8, 2008.
  87. 87.C. Chow, “On optimum recognition error and reject tradeoff,” IEEE Transactions on Information Theory, 1970.
  88. 88.G. Fumera and F. Roli, “Support vector machines with embedded reject option,” in International Workshop on Support Vector Machines, 2002.
  89. 89.C. Cortes and V. Vapnik, “Support-vector networks,” Machine learning, 1995.
  90. 90.A. Nguyen, J. Yosinski, and J. Clune, “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images,” in CVPR, 2015.
  91. 91.K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy, “Domain generalization: A survey,” arXiv preprint arXiv:2103.02503, 2021.
  92. 92.Z. Liu, Z. Miao, X. Pan, X. Zhan, D. Lin, S. X. Yu, and B. Gong, “Open compound domain adaptation,” in CVPR, 2020.
  93. 93.K. Han, A. Vedaldi, and A. Zisserman, “Learning to discover novel visual categories via deep transfer clustering,” in CVPR, 2019.
  94. 94.B. Zhao and K. Han, “Novel visual category discovery with dual ranking statistics and mutual knowledge distillation,” NeurIPS, 2021.
  95. 95.X. Jia, K. Han, Y. Zhu, and B. Green, “Joint representation learning and novel category discovery on single-and multi-modal data,” in ICCV, 2021.
  96. 96.S. Vaze, K. Han, A. Vedaldi, and A. Zisserman, “Generalized category discovery,” in CVPR, 2022.
  97. 97.K. Joseph, S. Paul, G. Aggarwal, S. Biswas, P. Rai, K. Han, and V. N. Balasubramanian, “Novel class discovery without forgetting,” in ECCV, 2022.
  98. 98.W. Wang, V. W. Zheng, H. Yu, and C. Miao, “A survey of zero-shot learning: Settings, methods, and applications,” TIST, 2019.
  99. 99.A. Bendale and T. Boult, “Towards open world recognition,” in CVPR, 2015.
  100. 100.Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu, “Large-scale long-tailed recognition in an open world,” in CVPR, 2019.
  101. 101.J. Parmar, S. Chouhan, V. Raychoudhury, and S. Rathore, “Open-world machine learning: applications, challenges, and opportunities,” ACM Computing Surveys, vol. 55, no. 10, pp. 1–37, 2023.
  102. 102.G. Shafer and V. Vovk, “A tutorial on conformal prediction.,” Journal of Machine Learning Research, vol. 9, no. 3, 2008.
  103. 103.A. N. Angelopoulos and S. Bates, “A gentle introduction to conformal prediction and distribution-free uncertainty quantification,” arXiv preprint arXiv:2107.07511, 2021.
  104. 104.R. Kaur, S. Jha, A. Roy, S. Park, E. Dobriban, O. Sokolsky, and I. Lee, “idecode: In-distribution equivariance for conformal out-of-distribution detection,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 7104–7114, 2022.
  105. 105.R. Kaur, K. Sridhar, S. Park, S. Jha, A. Roy, O. Sokolsky, and I. Lee, “Codit: Conformal out-of-distribution detection in time-series data,” arXiv e-prints, 2022.
  106. 106.F. Cai, A. I. Ozdagli, N. Potteiger, and X. Koutsoukos, “Inductive conformal out-of-distribution detection based on adversarial autoencoders,” in 2021 IEEE International Conference on Omni-Layer Intelligent Systems (COINS), pp. 1–6, IEEE, 2021.
  107. 107.M. Salehi, H. Mirzaei, D. Hendrycks, Y. Li, M. H. Rohban, and M. Sabokrou, “A unified survey on anomaly, novelty, open-set, and out-of-distribution detection: Solutions and future challenges,” arXiv preprint arXiv:2110.14051, 2021.
  108. 108.S. Liang, Y. Li, and R. Srikant, “Enhancing the reliability of out-of-distribution image detection in neural networks,” in ICLR, 2018.
  109. 109.K. Lee, K. Lee, H. Lee, and J. Shin, “A simple unified framework for detecting out-of-distribution samples and adversarial attacks,” in NeurIPS, 2018.
  110. 110.W. Liu, X. Wang, J. D. Owens, and Y. Li, “Energy-based out-of-distribution detection,” in NeurIPS, 2020.
  111. 111.C. S. Sastry and S. Oore, “Detecting out-of-distribution examples with gram matrices,” in ICML, 2020.
  112. 112.H. Wang, W. Liu, A. Bocchieri, and Y. Li, “Can multi-label classification networks know what they don’t know?,” NeurIPS, 2021.
  113. 113.J. Zhang, Q. Fu, X. Chen, L. Du, Z. Li, G. Wang, S. Han, D. Zhang, et al., “Out-of-distribution detection based on in-distribution data patterns memorization with modern hopfield energy,” in ICLR, 2023.
  114. 114.Y. Sun, C. Guo, and Y. Li, “React: Out-of-distribution detection with rectified activations,” in NeurIPS, 2021.
  115. 115.X. Dong, J. Guo, A. Li, W.-T. Ting, C. Liu, and H. Kung, “Neural mean discrepancy for efficient out-of-distribution detection,” in CVPR, 2022.
  116. 116.Y. Sun and Y. Li, “Dice: Leveraging sparsification for out-of-distribution detection,” in ECCV, 2022.
  117. 117.Y. Sun, Y. Ming, X. Zhu, and Y. Li, “Out-of-distribution detection with deep nearest neighbors,” in ICML, 2022.
  118. 118.Z. Lin, S. D. Roy, and Y. Li, “Mood: Multi-level out-of-distribution detection,” in CVPR, 2021.
  119. 119.C. S. Sastry and S. Oore, “Detecting out-of-distribution examples with in-distribution examples and gram matrices,” in NeurIPS-W, 2019.
  120. 120.A. Djurisic, N. Bozanic, A. Ashok, and R. Liu, “Extremely simple activation shaping for out-of-distribution detection,” ICLR, 2023.
  121. 121.J. Park, Y. G. Jung, and A. B. J. Teoh, “Nearest neighbor guidance for out-of-distribution detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1686–1695, 2023.
  122. 122.J. Park, J. C. L. Chai, J. Yoon, and A. B. J. Teoh, “Understanding the feature norm for out-of-distribution detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1557–1567, 2023.
  123. 123.X. Jiang, F. Liu, Z. Fang, H. Chen, T. Liu, F. Zheng, and B. Han, “Detecting out-of-distribution data through in-distribution class prior,” in International Conference on Machine Learning, pp. 15067–15088, PMLR, 2023.
  124. 124.X. Liu, Y. Lochman, and C. Zach, “Gen: Pushing the limits of softmax-based out-of-distribution detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23946–23955, 2023.
  125. 125.T. DeVries and G. W. Taylor, “Learning confidence for out-of-distribution detection in neural networks,” arXiv preprint arXiv:1802.04865, 2018.
  126. 126.Y. Wang, B. Li, T. Che, K. Zhou, Z. Liu, and D. Li, “Energy-based open-world uncertainty modeling for confidence calibration,” in ICCV, 2021.
  127. 127.A. Vyas, N. Jammalamadaka, X. Zhu, D. Das, B. Kaul, and T. L. Willke, “Out-of-distribution detection using an ensemble of self supervised leave-out classifiers,” in ECCV, 2018.
  128. 128.J. Bitterwolf, A. Meinke, and M. Hein, “Certifiably adversarially robust detection of out-of-distribution data,” in NeurIPS, 2020.
  129. 129.J. Chen, Y. Li, X. Wu, Y. Liang, and S. Jha, “Robust out-of-distribution detection for neural networks,” arXiv preprint arXiv:2003.09711, 2020.
  130. 130.M. Hein, M. Andriushchenko, and J. Bitterwolf, “Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem,” in CVPR, 2019.
  131. 131.S. Choi and S.-Y. Chung, “Novelty detection via blurring,” in ICLR, 2020.
  132. 132.J. Chen, Y. Li, X. Wu, Y. Liang, and S. Jha, “Atom: Robustifying out-of-distribution detection using outlier mining,” ECML&PKDD, 2021.
  133. 133.M. Hein, M. Andriushchenko, and J. Bitterwolf, “Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem,” in CVPR, 2019.
  134. 134.S. Thulasidasan, G. Chennupati, J. Bilmes, T. Bhattacharya, and S. Michalak, “On mixup training: Improved calibration and predictive uncertainty for deep neural networks,” in NeurIPS, 2019.
  135. 135.S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” in CVPR, 2019.
  136. 136.T. DeVries and G. W. Taylor, “Improved regularization of convolutional neural networks with cutout,” arXiv preprint arXiv:1708.04552, 2017.
  137. 137.D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan, “Augmix: A simple data processing method to improve robustness and uncertainty,” arXiv preprint arXiv:1912.02781, 2019.
  138. 138.D. Hendrycks, A. Zou, M. Mazeika, L. Tang, D. Song, and J. Steinhardt, “Pixmix: Dreamlike pictures comprehensively improve safety measures,” 2022.
  139. 139.J. Tack, S. Mo, J. Jeong, and J. Shin, “Csi: Novelty detection via contrastive learning on distributionally shifted instances,” in NeurIPS, 2020.
  140. 140.A. Meinke and M. Hein, “Towards neural networks that provably know when they don’t know,” arXiv preprint arXiv:1909.12180, 2019.
  141. 141.K. Bibas, M. Feder, and T. Hassner, “Single layer predictive normalized maximum likelihood for out-of-distribution detection,” NeurIPS, 2021.
  142. 142.Q. Wang, F. Liu, Y. Zhang, J. Zhang, C. Gong, T. Liu, and B. Han, “Watermarking for out-of-distribution detection,” in NeurIPS, 2022.
  143. 143.X. Dong, J. Guo, W.-T. T. Ang Li23, C. Liu, and H. Kung, “Neural mean discrepancy for efficient out-of-distribution detection,” in CVPR, 2022.
  144. 144.H. Wei, R. Xie, H. Cheng, L. Feng, B. An, and Y. Li, “Mitigating neural network overconfidence with logit normalization,” in ICML, 2022.
  145. 145.K. Lee, K. Lee, K. Min, Y. Zhang, J. Shin, and H. Lee, “Hierarchical novelty detection for visual object recognition,” in CVPR, 2018.
  146. 146.R. Huang and Y. Li, “Mos: Towards scaling out-of-distribution detection for large semantic space,” in CVPR, 2021.
  147. 147.R. Linderman, J. Zhang, N. Inkawhich, H. Li, and Y. Chen, “Fine-grain inference on out-of-distribution data with hierarchical classification,” in Proceedings of The 2nd Conference on Lifelong Learning Agents (S. Chandar, R. Pascanu, H. Sedghi, and D. Precup, eds.), vol. 232 of Proceedings of Machine Learning Research, pp. 162–183, PMLR, 22–25 Aug 2023.
  148. 148.G. Shalev, Y. Adi, and J. Keshet, “Out-of-distribution detection using multiple semantic label representations,” in NeurIPS, 2018.
  149. 149.S. Fort, J. Ren, and B. Lakshminarayanan, “Exploring the limits of out-of-distribution detection,” NeurIPS, 2021.
  150. 150.W. Gan, “Language guided out-of-distribution detection,” 2021.
  151. 151.Q. Yu and K. Aizawa, “Unsupervised out-of-distribution detection by maximum classifier discrepancy,” in ICCV, 2019.
  152. 152.S. Mohseni, M. Pitale, J. Yadawa, and Z. Wang, “Self-supervised learning for generalizable out-of-distribution detection,” in AAAI, 2020.
  153. 153.S. Thulasidasan, S. Thapa, S. Dhaubhadel, G. Chennupati, T. Bhattacharya, and J. Bilmes, “An effective baseline for robustness to distributional shift,” arXiv preprint arXiv:2105.07107, 2021.
  154. 154.A.-A. Papadopoulos, M. R. Rajati, N. Shaikh, and J. Wang, “Outlier exposure with confidence control for out-of-distribution detection,” Neurocomputing, 2021.
  155. 155.Y. Ming, Y. Fan, and Y. Li, “Poem: Out-of-distribution detection with posterior sampling,” in ICML, 2022.
  156. 156.Y. Li and N. Vasconcelos, “Background data resampling for outlier-aware classification,” in CVPR, 2020.
  157. 157.S. Mohseni, M. Pitale, J. Yadawa, and Z. Wang, “Self-supervised learning for generalizable out-of-distribution detection,” in AAAI, 2020.
  158. 158.J. Yang, H. Wang, L. Feng, X. Yan, H. Zheng, W. Zhang, and Z. Liu, “Semantically coherent out-of-distribution detection,” in ICCV, 2021.
  159. 159.F. Lu, K. Zhu, W. Zhai, K. Zheng, and Y. Cao, “Uncertainty-aware optimal transport for semantically coherent out-of-distribution detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3282–3291, 2023.
  160. 160.A. Shafaei, M. Schmidt, and J. J. Little, “A less biased evaluation of out-of-distribution sample detectors,” in BMVC, 2019.
  161. 161.J. Katz-Samuels, J. Nakhleh, R. Nowak, and Y. Li, “Training ood detectors in their natural habitats,” in International Conference on Machine Learning (ICML), PMLR, 2022.
  162. 162.Q. Wang, Z. Fang, Y. Zhang, F. Liu, Y. Li, and B. Han, “Learning to augment distributions for out-of-distribution detection,” arXiv preprint arXiv:2311.01796, 2023.
  163. 163.K. Lee, H. Lee, K. Lee, and J. Shin, “Training confidence-calibrated classifiers for detecting out-of-distribution samples,” 2018.
  164. 164.S. Vernekar, A. Gaurav, V. Abdelzad, T. Denouden, R. Salay, and K. Czarnecki, “Out-of-distribution detection in classifiers via generation,” in NeurIPS-W, 2019.
  165. 165.K. Sricharan and A. Srivastava, “Building robust classifiers through generation of confident out of distribution examples,” in NeurIPS-W, 2018.
  166. 166.T. Jeong and H. Kim, “Ood-maml: Meta-learning for few-shot out-of-distribution detection and classification,” in NeurIPS, 2020.
  167. 167.X. Du, Z. Wang, M. Cai, and Y. Li, “Vos: Learning what you don’t know by virtual outlier synthesis,” in Proceedings of the International Conference on Learning Representations, 2022.
  168. 168.L. Tao, X. Du, X. Zhu, and Y. Li, “Non-parametric outlier synthesis,” in ICLR, 2023.
  169. 169.Q. Wang, J. Ye, F. Liu, Q. Dai, M. Kalander, T. Liu, J. Hao, and B. Han, “Out-of-distribution detection with implicit outlier transformation,” 2023.
  170. 170.H. Zheng, Q. Wang, Z. Fang, X. Xia, F. Liu, T. Liu, and B. Han, “Out-of-distribution detection learning with unreliable out-of-distribution sources,” in NeurIPS, 2023.
  171. 171.X. Du, X. Wang, G. Gozum, and Y. Li, “Unknown-aware object detection: Learning what you don’t know from videos in the wild,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  172. 172.R. Huang, A. Geng, and Y. Li, “On the importance of gradients for detecting distributional shifts in the wild,” in NeurIPS, 2021.
  173. 173.C. Igoe, Y. Chung, I. Char, and J. Schneider, “How useful are gradients for ood detection really?,” arXiv preprint arXiv:2205.10439, 2022.
  174. 174.Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in ICML, 2016.
  175. 175.B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” NeurIPS, 2017.
  176. 176.K. Osawa, S. Swaroop, A. Jain, R. Eschenhagen, R. E. Turner, R. Yokota, and M. E. Khan, “Practical deep learning with bayesian principles,” in NeurIPS, 2019.
  177. 177.A. Malinin and M. Gales, “Predictive uncertainty estimation via prior networks,” in NeurIPS, 2018.
  178. 178.A. Malinin and M. Gales, “Reverse kl-divergence training of prior networks: Improved uncertainty and adversarial robustness,” in NeurIPS, 2019.
  179. 179.J. Nandy, W. Hsu, and M. L. Lee, “Towards maximizing the representation gap between in-domain & out-of-distribution examples,” in NeurIPS, 2020.
  180. 180.K. Kim, J. Shin, and H. Kim, “Locally most powerful bayesian test for out-of-distribution detection using deep generative models,” NeurIPS, 2021.
  181. 181.D. Hendrycks, K. Lee, and M. Mazeika, “Using pre-training can improve model robustness and uncertainty,” in International conference on machine learning, pp. 2712–2721, PMLR, 2019.
  182. 182.D. Hendrycks, X. Liu, E. Wallace, A. Dziedzic, R. Krishnan, and D. Song, “Pretrained transformers improve out-of-distribution robustness,” arXiv preprint arXiv:2004.06100, 2020.
  183. 183.Y. Ming and Y. Li, “How does fine-tuning impact out-of-distribution detection for vision-language models?,” IJCV, 2023.
  184. 184.A. Miyai, Q. Yu, G. Irie, and K. Aizawa, “Can pre-trained networks detect familiar out-of-distribution data?,” arXiv preprint arXiv:2310.00847, 2023.
  185. 185.A. Miyai, Q. Yu, G. Irie, and K. Aizawa, “Locoop: Few-shot out-of-distribution detection via prompt learning,” arXiv preprint arXiv:2306.01293, 2023.
  186. 186.F. Lu, K. Zhu, K. Zheng, W. Zhai, and Y. Cao, “Likelihood-aware semantic alignment for full-spectrum out-of-distribution detection,” arXiv preprint arXiv:2312.01732, 2023.
  187. 187.S. Esmaeilpour, B. Liu, E. Robertson, and L. Shu, “Zero-shot out-of-distribution detection based on the pretrained model clip,” in AAAI, 2022.
  188. 188.Y. Ming, Z. Cai, J. Gu, Y. Sun, W. Li, and Y. Li, “Delving into out-of-distribution detection with vision-language representations,” Advances in Neural Information Processing Systems, vol. 35, pp. 35087–35102, 2022.
  189. 189.H. Wang, Y. Li, H. Yao, and X. Li, “Clipn for zero-shot ood detection: Teaching clip to say no,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1802–1812, 2023.
  190. 190.B. Zong, Q. Song, M. R. Min, W. Cheng, C. Lumezanu, D. Cho, and H. Chen, “Deep autoencoding gaussian mixture model for unsupervised anomaly detection,” in ICLR, 2018.
  191. 191.D. Abati, A. Porrello, S. Calderara, and R. Cucchiara, “Latent space autoregression for novelty detection,” in CVPR, 2019.
  192. 192.S. Pidhorskyi, R. Almohsen, D. A. Adjeroh, and G. Doretto, “Generative probabilistic novelty detection with adversarial autoencoders,” in NeurIPS, 2018.
  193. 193.L. Deecke, R. Vandermeulen, L. Ruff, S. Mandt, and M. Kloft, “Image anomaly detection with generative adversarial networks,” in ECML&KDD, 2018.
  194. 194.M. Sabokrou, M. Khalooei, M. Fathy, and E. Adeli, “Adversarially learned one-class classifier for novelty detection,” in CVPR, 2018.
  195. 195.I. Kobyzev, S. Prince, and M. Brubaker, “Normalizing flows: An introduction and review of current methods,” TPAMI, 2020.
  196. 196.E. Zisselman and A. Tamar, “Deep residual flow for out of distribution detection,” in CVPR, 2020.
  197. 197.D. P. Kingma and P. Dhariwal, “Glow: Generative flow with invertible 1x1 convolutions,” NeurIPS, 2018.
  198. 198.A. Van Oord, N. Kalchbrenner, and K. Kavukcuoglu, “Pixel recurrent neural networks,” in ICML, 2016.
  199. 199.D. Jiang, S. Sun, and Y. Yu, “Revisiting flow generative models for out-of-distribution detection,” in International Conference on Learning Representations, 2021.
  200. 200.E. Nalisnick, A. Matsukawa, Y. W. Teh, D. Gorur, and B. Lakshminarayanan, “Do deep generative models know what they don’t know?,” in NeurIPS, 2018.
  201. 201.H. Choi, E. Jang, and A. A. Alemi, “Waic, but why? generative ensembles for robust anomaly detection,” arXiv preprint arXiv:1810.01392, 2018.
  202. 202.P. Kirichenko, P. Izmailov, and A. G. Wilson, “Why normalizing flows fail to detect out-of-distribution data,” in NeurIPS, 2020.
  203. 203.J. Ren, P. J. Liu, E. Fertig, J. Snoek, R. Poplin, M. A. DePristo, J. V. Dillon, and B. Lakshminarayanan, “Likelihood ratios for out-of-distribution detection,” in NeurIPS, 2019.
  204. 204.J. Serr`a, D. Alvarez, V. G´omez, O. Slizovskaia, ´ J. F. N´u˜nez, and J. Luque, “Input complexity and out-of-distribution detection with likelihood-based generative models,” 2020.
  205. 205.Z. Xiao, Q. Yan, and Y. Amit, “Likelihood regret: An out-of-distribution detection score for variational auto-encoder,” in NeurIPS, 2020.
  206. 206.H. Wang, Z. Li, L. Feng, and W. Zhang, “Vim: Out-of-distribution with virtual-logit matching,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  207. 207.J. Ren, S. Fort, J. Liu, A. G. Roy, S. Padhy, and B. Lakshminarayanan, “A simple fix to mahalanobis distance for improving near-ood detection,” arXiv preprint arXiv:2106.09022, 2021.
  208. 208.E. Techapanurak, M. Suganuma, and T. Okatani, “Hyperparameter-free out-of-distribution detection using cosine similarity,” in ACCV, 2020.
  209. 209.X. Chen, X. Lan, F. Sun, and N. Zheng, “A boundary based out-of-distribution classifier for generalized zero-shot learning,” in ECCV, 2020.
  210. 210.A. Zaeemzadeh, N. Bisagno, Z. Sambugaro, N. Conci, N. Rahnavard, and M. Shah, “Out-of-distribution detection using union of 1-dimensional subspaces,” in CVPR, 2021.
  211. 211.J. Van Amersfoort, L. Smith, Y. W. Teh, and Y. Gal, “Uncertainty estimation using a single deep deterministic neural network,” in ICML, 2020.
  212. 212.H. Huang, Z. Li, L. Wang, S. Chen, B. Dong, and X. Zhou, “Feature space singularity for out-of-distribution detection,” arXiv preprint arXiv:2011.14654, 2020.
  213. 213.Y. Ming, Y. Sun, O. Dia, and Y. Li, “Cider: Exploiting hyperspherical embeddings for out-of-distribution detection,” in ICLR, 2023.
  214. 214.J.-H. Kim, S. Yun, and H. O. Song, “Neural relation graph: A unified framework for identifying label noise and outlier data,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  215. 215.T. Denouden, R. Salay, K. Czarnecki, V. Abdelzad, B. Phan, and S. Vernekar, “Improving reconstruction autoencoder out-of-distribution detection with mahalanobis distance,” arXiv preprint arXiv:1812.02765, 2018.
  216. 216.Y. Zhou, “Rethinking reconstruction autoencoder-based out-of-distribution detection,” in CVPR, 2022.
  217. 217.Y. Yang, R. Gao, and Q. Xu, “Out-of-distribution detection with semantic mismatch under masking,” in ECCV, 2022.
  218. 218.W. Jiang, H. Cheng, M. Chen, S. Feng, Y. Ge, and C. Wang, “Read: Aggregating reconstruction error into out-of-distribution detection,” in AAAI, 2023.
  219. 219.J. Li, P. Chen, S. Yu, Z. He, S. Liu, and J. Jia, “Rethinking out-of-distribution (ood) detection: Masked image modeling is all you need,” in CVPR, 2023.
  220. 220.P. Morteza and Y. Li, “Provable guarantees for understanding out-of-distribution detection,” in AAAI, 2022.
  221. 221.L. P. Jain, W. J. Scheirer, and T. E. Boult, “Multi-class open set recognition using probability of inclusion,” in ECCV, 2014.
  222. 222.E. M. Rudd, L. P. Jain, W. J. Scheirer, and T. E. Boult, “The extreme value machine,” IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 3, pp. 762–768, 2017.
  223. 223.Z. Fang, Y. Li, J. Lu, J. Dong, B. Han, and F. Liu, “Is out-of-distribution detection learnable?,” in NeurIPS, 2022.
  224. 224.A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., “Learning transferable visual models from natural language supervision,” in ICML, 2021.
  225. 225.E. T. Jaynes, “Bayesian methods: General background,” 1986.
  226. 226.R. M. Neal, Bayesian learning for neural networks. 2012.
  227. 227.D. Gamerman and H. F. Lopes, Markov chain Monte Carlo: stochastic simulation for Bayesian inference. CRC Press, 2006.
  228. 228.D. J. C. Mackay, Bayesian methods for adaptive models. PhD thesis, California Institute of Technology, 1992.
  229. 229.A. Y. Foong, Y. Li, J. M. Hern´andez-Lobato, and R. E. Turner, “’in-between’ uncertainty in bayesian neural networks,” in ICML-W, 2020.
  230. 230.C. Peterson and E. Hartman, “Explorations of the mean field theory learning algorithm,” Neural Networks, 1989.
  231. 231.F. Wenzel, K. Roth, B. S. Veeling, J. Swiatkowski, ´ L. Tran, S. Mandt, J. Snoek, T. Salimans, R. Jenatton, and S. Nowozin, “How good is the bayes posterior in deep neural networks really?,” in ICML, 2020.
  232. 232.A. Gelman, “Objections to bayesian statistics,” Bayesian Analysis, 2008.
  233. 233.T. G. Dietterich, “Ensemble methods in machine learning,” in International workshop on multiple classifier systems, 2000.
  234. 234.W. J. Maddox, P. Izmailov, T. Garipov, D. P. Vetrov, and A. G. Wilson, “A simple baseline for bayesian uncertainty in deep learning,” Advances in Neural Information Processing Systems, vol. 32, pp. 13153–13164, 2019.
  235. 235.R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al., “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258, 2021.
  236. 236.K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Learning to prompt for vision-language models,” International Journal of Computer Vision (IJCV), 2022.
  237. 237.K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Conditional prompt learning for vision-language models,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  238. 238.M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, “Visual prompt tuning,” in European Conference on Computer Vision, pp. 709–727, Springer, 2022.
  239. 239.P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang, Y. Zhang, H. Li, and Y. Qiao, “Clip-adapter: Better vision-language models with feature adapters,” International Journal of Computer Vision, pp. 1–15, 2023.
  240. 240.J. Yang, K. Zhou, and Z. Liu, “Full-spectrum out-of-distribution detection,” arXiv preprint arXiv:2204.05306, 2022.
  241. 241.E. D. C. Gomes, F. Alberge, P. Duhamel, and P. Piantanida, “Igeood: An information geometry approach to out-of-distribution detection,” in ICLR, 2022.
  242. 242.W. J. Scheirer, L. P. Jain, and T. E. Boult, “Probability models for open set recognition,” TPAMI, 2014.
  243. 243.R. L. Smith, “Extreme value theory,” Handbook of applicable mathematics, 1990.
  244. 244.E. Castillo, Extreme value theory in engineering. Elsevier, 2012.
  245. 245.A. Bendale and T. E. Boult, “Towards open set deep networks,” in CVPR, 2016.
  246. 246.P. Perera and V. M. Patel, “Deep transfer learning for multiple class novelty detection,” in CVPR, 2019.
  247. 247.P. Perera, V. I. Morariu, R. Jain, V. Manjunatha, C. Wigington, V. Ordonez, and V. M. Patel, “Generative-discriminative feature representations for open-set recognition,” in CVPR, 2020.
  248. 248.X. Sun, H. Ding, C. Zhang, G. Lin, and K.-V. Ling, “M2iosr: Maximal mutual information open set recognition,” arXiv preprint arXiv:2108.02373, 2021.
  249. 249.Z. Ge, S. Demyanov, Z. Chen, and R. Garnavi, “Generative openmax for multi-class open set classification,” in BMVC, 2017.
  250. 250.L. Neal, M. Olson, X. Fern, W.-K. Wong, and F. Li, “Open set learning with counterfactual images,” in ECCV, 2018.
  251. 251.D.-W. Zhou, H.-J. Ye, and D.-C. Zhan, “Learning placeholders for open-set recognition,” in CVPR, 2021.
  252. 252.S. Kong and D. Ramanan, “Opengan: Open-set recognition via open data generation,” in ICCV, 2021.
  253. 253.C. Geng and S. Chen, “Collective decision for open set recognition,” TKDE, 2020.
  254. 254.J. Jang and C. O. Kim, “One-vs-rest network-based deep probability model for open set recognition,” arXiv preprint arXiv:2004.08067, 2020.
  255. 255.P. Schlachter, Y. Liao, and B. Yang, “Open-set recognition using intra-class splitting,” in EUSIPCO, 2019.
  256. 256.M. Masana, I. Ruiz, J. Serrat, J. van de Weijer, and A. M. Lopez, “Metric learning for novelty and anomaly detection,” in BMVC, 2018.
  257. 257.Y. Shu, Y. Shi, Y. Wang, T. Huang, and Y. Tian, “p-odn: prototype-based open deep network for open set recognition,” Scientific reports, 2020.
  258. 258.B. Liu, H. Kang, H. Li, G. Hua, and N. Vasconcelos, “Few-shot open-set recognition using meta-learning,” in CVPR, 2020.
  259. 259.G. Chen, L. Qiao, Y. Shi, P. Peng, J. Li, T. Huang, S. Pu, and Y. Tian, “Learning open set network with discriminative reciprocal points,” in ECCV, 2020.
  260. 260.R. Yoshihashi, W. Shao, R. Kawakami, S. You, M. Iida, and T. Naemura, “Classification-reconstruction learning for open-set recognition,” in CVPR, 2019.
  261. 261.A. Cao, Y. Luo, and D. Klabjan, “Open-set recognition with gaussian mixture variational autoencoders,” AAAI, 2020.
  262. 262.P. R. M. J´unior, R. M. De Souza, R. d. O. Werneck, B. V. Stein, D. V. Pazinato, W. R. de Almeida, O. A. Penatti, R. d. S. Torres, and A. Rocha, “Nearest neighbors distance ratio open-set classifier,” Machine Learning, 2017.
  263. 263.H. Zhang and V. M. Patel, “Sparse representation-based open set recognition,” TPAMI, 2016.
  264. 264.P. Bodesheim, A. Freytag, E. Rodner, M. Kemmler, and J. Denzler, “Kernel null space methods for novelty detection,” in CVPR, 2013.
  265. 265.J. Liu, Z. Lian, Y. Wang, and J. Xiao, “Incremental kernel null space discriminant analysis for novelty detection,” in CVPR, 2017.
  266. 266.P. Oza and V. M. Patel, “C2ae: Class conditioned auto-encoder for open-set recognition,” in CVPR, 2019.
  267. 267.X. Sun, Z. Yang, C. Zhang, K.-V. Ling, and G. Peng, “Conditional gaussian distribution learning for open set recognition,” in CVPR, 2020.
  268. 268.Z. Yue, T. Wang, Q. Sun, X.-S. Hua, and H. Zhang, “Counterfactual zero-shot and open-set visual recognition,” in CVPR, 2021.
  269. 269.R. Shao, P. Perera, P. C. Yuen, and V. M. Patel, “Open-set adversarial defense,” in ECCV, 2020.
  270. 270.H. Zhang, A. Li, J. Guo, and Y. Guo, “Hybrid models for open set recognition,” in ECCV, 2020.
  271. 271.S. Vaze, K. Han, A. Vedaldi, and A. Zisserman, “Open-set recognition: A good closed-set classifier is all you need,” in ICLR, 2022.
  272. 272.G. Danuser and M. Stricker, “Parametric model fitting: From inlier characterization to outlier detection,” TPAMI, 1998.
  273. 273.R. De Maesschalck, D. Jouan-Rimbaud, and D. L. Massart, “The mahalanobis distance,” Chemometrics and intelligent laboratory systems, 2000.
  274. 274.C. Leys, O. Klein, Y. Dominicy, and C. Ley, “Detecting multivariate outliers: Use a robust variant of the mahalanobis distance,” Journal of Experimental Social Psychology, 2018.
  275. 275.R. A. Redner and H. F. Walker, “Mixture densities, maximum likelihood and the em algorithm,” SIAM review, 1984.
  276. 276.E. Eskin, “Anomaly detection over noisy data using learned probability distributions,” in ICML, 2000.
  277. 277.M. Turcotte, J. Moore, N. Heard, and A. McPhall, “Poisson factorization for peer-based anomaly detection,” in IEEE Conference on Intelligence and Security Informatics (ISI), 2016.
  278. 278.A. J. Izenman, “Review papers: Recent developments in nonparametric density estimation,” Journal of the American Statistical Association, 1991.
  279. 279.J. Van Ryzin, “A histogram method of density estimation,” Communications in Statistics-Theory and Methods, 1973.
  280. 280.M. Xie, J. Hu, and B. Tian, “Histogram-based online anomaly detection in hierarchical wireless sensor networks,” in ICTSPCC, 2012.
  281. 281.A. Kind, M. P. Stoecklin, and X. Dimitropoulos, “Histogram-based traffic anomaly detection,” IEEE Transactions on Network and Service Management, 2009.
  282. 282.M. Goldstein and A. Dengel, “Histogram-based outlier score (hbos): A fast unsupervised anomaly detection algorithm,” KI-2012: Poster and Demo Track, 2012.
  283. 283.E. Parzen, “On estimation of a probability density function and mode,” The annals of mathematical statistics, 1962.
  284. 284.M. Desforges, P. Jacob, and J. Cooper, “Applications of probability density estimation to the detection of abnormal conditions in engineering,” Proceedings of the institution of mechanical engineers, 1998.
  285. 285.W. Hu, J. Gao, B. Li, O. Wu, J. Du, and S. Maybank, “Anomaly detection using local kernel density estimation and context-based regression,” TKDE, 2018.
  286. 286.M. A. Kramer, “Nonlinear principal component analysis using autoassociative neural networks,” AIChE journal, 1991.
  287. 287.D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114, 2013.
  288. 288.I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” NIPS, 2014.
  289. 289.D. Rezende and S. Mohamed, “Variational inference with normalizing flows,” in ICML, 2015.
  290. 290.J. Ngiam, Z. Chen, P. W. Koh, and A. Y. Ng, “Learning deep energy models,” in ICML, 2011.
  291. 291.S. Zhai, Y. Cheng, W. Lu, and Z. Zhang, “Deep structured energy based models for anomaly detection,” in ICML, 2016.
  292. 292.A. Hyv¨arinen and P. Dayan, “Estimation of non-normalized statistical models by score matching.,” 2005.
  293. 293.M. Welling and Y. W. Teh, “Bayesian learning via stochastic gradient langevin dynamics,” in ICML, 2011.
  294. 294.H. Wang, X. Wu, Z. Huang, and E. P. Xing, “High-frequency component helps explain the generalization of convolutional neural networks,” in CVPR, 2020.
  295. 295.G. Chen, P. Peng, L. Ma, J. Li, L. Du, and Y. Tian, “Amplitude-phase recombination: Rethinking robustness of convolutional neural networks in frequency domain,” ICCV, 2021.
  296. 296.H. Liu, X. Li, W. Zhou, Y. Chen, Y. He, H. Xue, W. Zhang, and N. Yu, “Spatial-phase shallow learning: rethinking face forgery detection in frequency domain,” in CVPR, 2021.
  297. 297.A. Adler, M. Elad, Y. Hel-Or, and E. Rivlin, “Sparse coding with anomaly detection,” Journal of Signal Processing Systems, 2015.
  298. 298.A. Li, Z. Miao, Y. Cen, and Y. Cen, “Anomaly detection using sparse reconstruction in crowded scenes,” Multimedia Tools and Applications, 2017.
  299. 299.X. Mo, V. Monga, R. Bala, and Z. Fan, “Adaptive sparse representations for video anomaly detection,” IEEE Transactions on Circuits and Systems for Video Technology, 2013.
  300. 300.Y. Xiao, H. Wang, W. Xu, and J. Zhou, “L1 norm based kpca for novelty detection,” Pattern Recognition, 2013.
  301. 301.K. Jiang, W. Xie, J. Lei, T. Jiang, and Y. Li, “Lren: Low-rank embedded network for sample-free hyperspectral anomaly detection,” in AAAI, 2021.
  302. 302.Z. Chen, C. K. Yeo, B. S. Lee, and C. T. Lau, “Autoencoder-based network anomaly detection,” in Wireless Telecommunications Symposium, 2018.
  303. 303.J. An and S. Cho, “Variational autoencoder based anomaly detection using reconstruction probability,” Special Lecture on IE, 2015.
  304. 304.H. Zenati, C. S. Foo, B. Lecouat, G. Manek, and V. R. Chandrasekhar, “Efficient gan-based anomaly detection,” in ICLR-W, 2018.
  305. 305.W. Liu, W. Luo, D. Lian, and S. Gao, “Future frame prediction for anomaly detection–a new baseline,” in CVPR, 2018.
  306. 306.D. Gong, L. Liu, V. Le, B. Saha, M. R. Mansour, S. Venkatesh, and A. v. d. Hengel, “Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection,” in CVPR, 2019.
  307. 307.H. Park, J. Noh, and B. Ham, “Learning memory-guided normality for anomaly detection,” in CVPR, 2020.
  308. 308.C.-H. Lai, D. Zou, and G. Lerman, “Robust subspace recovery layer for unsupervised anomaly detection,” ICLR, 2020.
  309. 309.X. Yan, H. Zhang, X. Xu, X. Hu, and P.-A. Heng, “Learning semantic context from normal samples for unsupervised anomaly detection,” in AAAI, 2021.
  310. 310.D. T. Nguyen, Z. Lou, M. Klar, and T. Brox, “Anomaly detection with multiple-hypotheses predictions,” in ICML, 2019.
  311. 311.K. Tian, S. Zhou, J. Fan, and J. Guan, “Learning competitive and discriminative reconstructions for anomaly detection,” in AAAI, 2019.
  312. 312.X. Han, X. Chen, and L.-P. Liu, “Gan ensemble for anomaly detection,” arXiv preprint arXiv:2012.07988, 2020.
  313. 313.G. Kwon, M. Prabhushankar, D. Temel, and G. AlRegib, “Backpropagated gradient representations for anomaly detection,” in ECCV, 2020.
  314. 314.D. Wettschereck, “A study of distance-based machine learning algorithms,” 1994.
  315. 315.J. Tian, M. H. Azarian, and M. Pecht, “Anomaly detection using self-organizing maps-based k-nearest neighbor algorithm,” in PHM Society European Conference, 2014.
  316. 316.G. M¨unz, S. Li, and G. Carle, “Traffic anomaly detection using k-means clustering,” in GI/ITG Workshop MMBnet, 2007.
  317. 317.I. Syarif, A. Prugel-Bennett, and G. Wills, “Unsupervised clustering approach for network anomaly detection,” in International conference on networked digital technologies, 2012.
  318. 318.D. M. J. Tax, “One-class classification: Concept learning in the absence of counter-examples.,” 2002.
  319. 319.L. Ruff, R. Vandermeulen, N. Goernitz, L. Deecke, S. A. Siddiqui, A. Binder, E. M¨uller, and M. Kloft, “Deep one-class classification,” in ICML, 2018.
  320. 320.B. Zhang and W. Zuo, “Learning from positive and unlabeled examples: A survey,” in International Symposiums on Information Processing, 2008.
  321. 321.J. Bekker and J. Davis, “Learning from positive and unlabeled data: A survey,” Machine Learning, 2020.
  322. 322.K. Jaskie and A. Spanias, “Positive and unlabeled learning algorithms and applications: A survey,” in International Conference on Information, Intelligence, Systems and Applications, 2019.
  323. 323.L. Ruff, R. A. Vandermeulen, N. G¨ornitz, A. Binder, E. M¨uller, K.-R. M¨uller, and M. Kloft, “Deep semi-supervised anomaly detection,” ICLR, 2020.
  324. 324.L. Bergman and Y. Hoshen, “Classification-based anomaly detection for general data,” in ICLR, 2020.
  325. 325.I. Golan and R. El-Yaniv, “Deep anomaly detection using geometric transformations,” in NeurIPS, 2018.
  326. 326.M.-I. Georgescu, A. Barbalau, R. T. Ionescu, F. S. Khan, M. Popescu, and M. Shah, “Anomaly detection in video via self-supervised and multi-task learning,” in CVPR, 2021.
  327. 327.D. G. Altman and J. M. Bland, “Standard deviations and standard errors,” BMJ, 2005.
  328. 328.C. Leys, C. Ley, O. Klein, P. Bernard, and L. Licata, “Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median,” Journal of experimental social psychology, 2013.
  329. 329.X. Yang, L. J. Latecki, and D. Pokrajac, “Outlier detection with globally optimal exemplar-based gmm,” in SIAM, 2009.
  330. 330.M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander, “Lof: identifying density-based local outliers,” in SIGMOD, 2000.
  331. 331.M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” Communications of the ACM, 1981.
  332. 332.M. Sugiyama and K. Borgwardt, “Rapid distance-based outlier detection via sampling,” NIPS, 2013.
  333. 333.G. H. Orair, C. H. Teixeira, W. Meira Jr, Y. Wang, and S. Parthasarathy, “Distance-based outlier detection: consolidation and renewed bearing,” Proceedings of the VLDB Endowment, 2010.
  334. 334.M. Ester, H.-P. Kriegel, J. Sander, X. Xu, et al., “A density-based algorithm for discovering clusters in large spatial databases with noise,” in KDD, 1996.
  335. 335.V. Hautamaki, I. Karkkainen, and P. Franti, “Outlier detection using k-nearest neighbour graph,” in ICPR, 2004.
  336. 336.F. Muhlenbach, S. Lallich, and D. A. Zighed, “Identifying and handling mislabelled instances,” Journal of Intelligent Information Systems, 2004.
  337. 337.W. Liu, J. He, and S.-F. Chang, “Large graph construction for scalable semi-supervised learning,” in ICML, 2010.
  338. 338.L. Akoglu, H. Tong, and D. Koutra, “Graph based anomaly detection and description: a survey,” Data mining and knowledge discovery, 2015.
  339. 339.C. C. Noble and D. J. Cook, “Graph-based anomaly detection,” in SIGKDD, 2003.
  340. 340.Y. Kou, C.-T. Lu, and R. F. Dos Santos, “Spatial outlier detection: a graph-based approach,” in 19th IEEE International Conference on Tools with Artificial Intelligence (ICTAI), 2007.
  341. 341.Z. Mingqiang, H. Hui, and W. Qian, “A graph-based clustering algorithm for anomaly intrusion detection,” in International Conference on Computer Science & Education (ICCSE), 2012.
  342. 342.Z.-F. Wu, T. Wei, J. Jiang, C. Mao, M. Tang, and Y.-F. Li, “Ngc: A unified framework for learning with open-world noisy data,” in ICCV, 2021.
  343. 343.J. Yang, W. Chen, L. Feng, X. Yan, H. Zheng, and W. Zhang, “Webly supervised image classification with metadata: Automatic noisy label correction via visual-semantic graph,” in ACM Multimedia, 2020.
  344. 344.F. T. Liu, K. M. Ting, and Z.-H. Zhou, “Isolation forest,” in ICDM, 2008.
  345. 345.Y. Li, J. Yang, Y. Song, L. Cao, J. Luo, and L.-J. Li, “Learning from noisy labels with distillation,” in CVPR, 2017.
  346. 346.D. T. Nguyen, C. K. Mummadi, T. P. N. Ngo, T. H. P. Nguyen, L. Beggel, and T. Brox, “Self: Learning to filter noisy labels with self-ensembling,” in ICLR, 2020.
  347. 347.B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama, “Co-teaching: Robust training of deep neural networks with extremely noisy labels,” in NIPS, 2018.
  348. 348.J. Yang, L. Feng, W. Chen, X. Yan, H. Zheng, P. Luo, and W. Zhang, “Webly supervised image classification with self-contained confidence,” in ECCV, 2020.
  349. 349.J. Yang, P. Wang, D. Zou, Z. Zhou, K. Ding, W. Peng, H. Wang, G. Chen, B. Li, Y. Sun, X. Du, K. Zhou, W. Zhang, D. Hendrycks, Y. Li, and Z. Liu, “Openood: Benchmarking generalized out-of-distribution detection,” in NeurIPS, 2022.
  350. 350.A. Krizhevsky, G. Hinton, et al., “Learning multiple layers of features from tiny images,” 2009.
  351. 351.A. Krizhevsky, V. Nair, and G. Hinton, “Cifar-10 and cifar-100 datasets,” URl: https://www. cs. toronto. edu/kriz/cifar. html, vol. 6, no. 1, p. 1, 2009.
  352. 352.Y. LeCun and C. Cortes, “The mnist database of handwritten digits,” 2005.
  353. 353.Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” 2011.
  354. 354.G. Kylberg, “Kylberg texture dataset v. 1.0,” 2011.
  355. 355.B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, “Places: A 10 million image database for scene recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
  356. 356.A. Torralba, R. Fergus, and W. T. Freeman, “80 million tiny images: A large data set for nonparametric object and scene recognition,” TPAMI, 2008.
  357. 357.K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, 2016.
  358. 358.J. Zhang, J. Yang, P. Wang, H. Wang, Y. Lin, H. Zhang, Y. Sun, X. Du, K. Zhou, W. Zhang, Y. Li, Z. Liu, Y. Chen, and H. Li, “Openood v1.5: Enhanced benchmark for out-of-distribution detection,” arXiv preprint arXiv:2306.09301, 2023.
  359. 359.J. Bitterwolf, M. M¨uller, and M. Hein, “In or out? fixing imagenet out-of-distribution detection evaluation,” in ICML, 2023.
  360. 360.P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. L. Phillips, I. Gao, et al., “Wilds: A benchmark of in-the-wild distribution shifts,” in International Conference on Machine Learning, pp. 5637–5664, PMLR, 2021.
  361. 361.L. Cultrera, L. Seidenari, and A. Del Bimbo, “Leveraging visual attention for out-of-distribution detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4447–4456, 2023.
  362. 362.Y. Ming, H. Yin, and Y. Li, “On the impact of spurious correlation for out-of-distribution detection,” in AAAI, 2022.
  363. 363.P. Panareda Busto and J. Gall, “Open set domain adaptation,” in ICCV, 2017.
  364. 364.Y. Shu, Z. Cao, C. Wang, J. Wang, and M. Long, “Open domain generalization with domain-augmented meta-learning,” in CVPR, 2021.
  365. 365.J. Li, C. Xiong, and S. C. Hoi, “Mopro: Webly supervised learning with momentum prototypes,” ICLR, 2021.
  366. 366.V. D. Nguyen, “Out-of-distribution detection for lidar-based 3d object detection,” Master’s thesis, University of Waterloo, 2022.
  367. 367.G. Shalev, G.-L. Shalev, and J. Keshet, “A baseline for detecting out-of-distribution examples in image captioning,” arXiv preprint arXiv:2207.05418, 2022.
  368. 368.X. Wu, J. Lu, Z. Fang, and G. Zhang, “Meta ood learning for continuously adaptive ood detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19353–19364, 2023.
  369. 369.C. Zhou, G. Neubig, J. Gu, M. Diab, P. Guzman, L. Zettlemoyer, and M. Ghazvininejad, “Detecting hallucinated content in conditional neural sequence generation,” ACL, 2020.
  370. 370.Y. Dai, H. Lang, K. Zeng, F. Huang, and Y. Li, “Exploring large language models for multi-modal out-of-distribution detection,” arXiv preprint arXiv:2310.08027, 2023.
  371. 371.Z. Yang, L. Li, K. Lin, J. Wang, C.-C. Lin, Z. Liu, and L. Wang, “The dawn of lmms: Preliminary explorations with gpt-4v (ision),” arXiv preprint arXiv:2309.17421, vol. 9, no. 1, 2023.
  372. 372.H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” arXiv preprint arXiv:2304.08485, 2023.
  373. 373.B. Li, Y. Zhang, L. Chen, J. Wang, J. Yang, and Z. Liu, “Otter: A multi-modal model with in-context instruction tuning,” arXiv preprint arXiv:2305.03726, 2023.

Citation

MLA
Yang, J., et al. “Generalized Out-of-Distribution Detection: A Survey”. arXiv, 2021, http://arxiv.org/abs/2110.11334v3.
APA
Yang, J., Zhou, K., Li, Y., & Liu, Z. (2021). Generalized Out-of-Distribution Detection: A Survey. arXiv. http://arxiv.org/abs/2110.11334v3
Chicago
Yang, J., K. Zhou, Y. Li, and Z. Liu. 2021. “Generalized Out-of-Distribution Detection: A Survey”. arXiv. http://arxiv.org/abs/2110.11334v3.
Harvard
Yang, J. et al. (2021) “Generalized Out-of-Distribution Detection: A Survey”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2110.11334v3.
Vancouver
1. Yang J, Zhou K, Li Y, Liu Z (2021) Generalized Out-of-Distribution Detection: A Survey. arXiv

BibTeX

@article{yang2021generalized,
  title = {Generalized Out-of-Distribution Detection: A Survey},
  author = {Yang, Jingkang and Zhou, Kaiyang and Li, Yixuan and Liu, Ziwei},
  year = {2021},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2110.11334v3},
  eprint = {2110.11334}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF