Back

Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss

Kaidi CaoColin WeiAdrien GaidonNikos AréchigaTengyu Ma

article2019NeurIPS2,224 citations

Proposes a theoretically grounded label-distribution-aware margin loss paired with a deferred re-weighting training schedule to substantially improve deep learning generalization on minority classes in heavily imbalanced datasets.

Listen

Real-world datasets for machine learning applications often suffer from extreme class imbalance, where a few frequent classes dominate while many critical minority classes contain very few examples. Standard deep neural networks trained on such data tend to perform poorly on rare categories. This poses serious risks in practical deployments, including autonomous systems, medical screening, and visual recognition, where balanced accuracy across all classes or reliable performance on rare cases is required. Existing remedies—such as re-weighting loss values or re-sampling minority examples—frequently cause models to severely overfit rare classes or destabilize optimization during training.

The article sets out to design and evaluate a theoretically grounded training loss and optimization strategy that boosts generalization on minority classes without sacrificing accuracy on frequent classes. Specifically, the authors aimed to demonstrate that enforcing class-dependent margins combined with a deferred re-weighting schedule outperforms existing class-imbalance techniques across standard benchmarks.

The researchers developed two complementary techniques: the Label-Distribution-Aware Margin loss, which mathematically enforces wider classification margins for rarer classes based on generalization error bounds, and a Deferred Re-balancing training schedule, which trains the model with standard empirical risk minimization in early epochs before applying class re-weighting in later epochs with a smaller learning rate. They validated this framework across controlled vision benchmarks, including imbalanced variations of standard image datasets (CIFAR-10, CIFAR-100, and Tiny ImageNet), an imbalanced text sentiment dataset, and the large-scale real-world iNaturalist 2018 dataset containing over 437,000 images across 8,142 categories.

The experimental findings show that the proposed approach substantially improves model accuracy across all test scenarios. On the large-scale iNaturalist benchmark, combining the new loss and deferred schedule achieved a top-1 error rate of 32.00%, outperforming the standard training baseline by 10.86 percentage points and prior state-of-the-art re-weighting methods by 6.88 percentage points. On vision datasets with severe 100-to-1 class imbalances, the framework consistently reduced classification error compared to standard baselines, dropping top-1 error from roughly 29.6% to 23.0% on 10-class image data and from 61.7% to 58.0% on 100-class data. Feature analysis demonstrated that deferring re-weighting allows networks to first build high-quality internal representations before re-balancing class decision boundaries, avoiding the severe under-fitting on frequent classes or overfitting on rare classes that plagues early re-weighting.

These results indicate that organizations deploying machine learning on imbalanced operational data can achieve major performance gains with minimal engineering overhead. Because the proposed loss and scheduling replace the standard loss function and optimization timetable, they do not introduce substantial computational complexity or inference latency. Furthermore, the framework generalizes effectively to domain-adaptation settings where target test class distributions are non-uniform but known.

Engineering and data science teams addressing class imbalance should adopt the label-distribution-aware loss in place of standard cross-entropy objectives and implement deferred re-balancing rather than weighting samples from the initial epoch. Organizations should evaluate this pipeline on internal imbalanced datasets through pilot experiments to calibrate the single margin hyperparameter against standard validation targets.

While empirical results strongly support the framework across multiple vision and language datasets, the theoretical derivation for optimal margin scaling assumes a binary classification boundary and separable data, serving as a heuristic approximation for complex multi-class settings. Additionally, the approach assumes that either the target evaluation distribution is balanced or its label proportions are known in advance, meaning caution is warranted if target domain class distributions are entirely unknown or dynamically shifting.

Stanford UniversityToyota Research Institutedeep learningout-of-distribution generalizationlabel-distribution-aware margin lossLDAM lossclass imbalance
arXiv: 1906.07413kaidic/LDAM-DRW
  • Paper: Class-Balanced Loss Based on Effective Number of Samples, Yin Cui et al. (2019). Reading this paper first is essential because it introduces the foundational effective-number-of-samples weighting perspective that directly motivates the margin-based loss framework of the source.
  • Paper: Focal Loss for Dense Object Detection, Tsung-Yi Lin et al. (2017). This work establishes the core concept of modulating loss functions to handle extreme class imbalance, providing the necessary precursor rationale for the margin adjustments explored in the source.
  • Paper: Supervised Contrastive Learning, Prannay Khosla et al. (2020). This book chapter naturally continues the source's exploration of representation learning and loss engineering by demonstrating how supervised contrastive objectives further improve margin-based classification.
  • Paper: Improved Baselines with Momentum Contrastive Learning, Xinlei Chen et al. (2020). This paper builds directly upon the foundational representation and loss techniques discussed in the source, extending them to improved unsupervised contrastive learning baselines.
Cover for Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss

Abstract

Deep learning algorithms can fare poorly when the training dataset suffers from heavy class-imbalance but the testing criterion requires good generalization on less frequent classes. We design two novel methods to improve performance in such scenarios. First, we propose a theoretically-principled label-distribution-aware margin (LDAM) loss motivated by minimizing a margin-based generalization bound. This loss replaces the standard cross-entropy objective during training and can be applied with prior strategies for training with class-imbalance such as re-weighting or re-sampling. Second, we propose a simple, yet effective, training schedule that defers re-weighting until after the initial stage, allowing the model to learn an initial representation while avoiding some of the complications associated with re-weighting or re-sampling. We test our methods on several benchmark vision tasks including the real-world imbalanced dataset iNaturalist 2018. Our experiments show that either of these methods alone can already improve over existing techniques and their combination achieves even better performance gains.

Topics

deep learningout-of-distribution generalization

Keywords

label-distribution-aware margin lossLDAM lossclass imbalancemargin-based generalizationempirical risk minimizationcross-entropy lossclass-conditional distributionminority classes

Table of Contents

  • 1 Introduction
  • 2 Related Works
  • 3 Main Approach
  • 3.1 Theoretical Motivations
  • 3.2 Label-Distribution-Aware Margin Loss
  • 3.3 Deferred Re-balancing Optimization Schedule
  • 4 Experiments
  • 4.1 Experimental results on IMDB review dataset
  • 4.2 Experimental results on CIFAR
  • 4.3 Visual recognition on iNaturalist 2018 and imbalanced Tiny ImageNet
  • 4.4 Ablation study
  • 5 Conclusion
  • References
  • A Missing Proofs and Derivations in Section
  • B Implementation details
  • C Additional Results
  • C.1 Feature visualization
  • C.2 Visual Recognition on imbalanced Tiny ImageNet
  • C.3 Comparing feature extractors trained by different schemes
  • C.4 Comparing DRW and DRS
  • C.5 Imbalanced Test Label Distributions

Knowls

  1. Knowl 1 — Label-Distribution-Aware Margin Loss

    model/method

    The Label-Distribution-Aware Margin (LDAM) loss is a classification loss function designed for training deep neural networks under severe class imbalance. For an input example x∈Rdx \in \mathbb{R}^d with class label y∈{1,…,k}y \in \{1, \dots, k\}, let z=f(x)∈Rkz = f(x) \in \mathbb{R}^k denote the predicted logit vector. The LDAM loss is defined as:

    LLDAM((x,y);f)=−log⁡ezy−Δyezy−Δy+∑j≠yezj\mathcal{L}_{\text{LDAM}}((x, y); f) = -\log \frac{e^{z_y - \Delta_y}}{e^{z_y - \Delta_y} + \sum_{j \neq y} e^{z_j}}

    where each class j∈{1,…,k}j \in \{1, \dots, k\} is assigned a class-dependent margin offset Δj\Delta_j inversely proportional to the fourth root of its training sample size njn_j:

    Δj=Cnj1/4\Delta_j = \frac{C}{n_j^{1/4}}

    where C>0C > 0 is a hyperparameter chosen to scale max⁡jΔj\max_j \Delta_j to a target maximum margin (e.g., 0.50.5).

    To ensure margin enforceability, the penultimate representation h(x)h(x) and each class weight vector WjW_j of the final classification layer are normalized to unit ℓ2\ell_2 norm before logit computation: zj=Wj⊤h(x)/∥h(x)∥2z_j = W_j^\top h(x) / \|h(x)\|_2.

    The non-smooth hinge variant of the loss is:

    LLDAM-HG((x,y);f)=max⁡(max⁡j≠y{zj}−zy+Δy,0)\mathcal{L}_{\text{LDAM-HG}}((x, y); f) = \max\left( \max_{j \neq y} \{z_j\} - z_y + \Delta_y, 0 \right)

  2. Knowl 2 — Per-Class and Balanced Margin-Based Generalization Bounds

    theoretical result

    Let F\mathcal{F} be a hypothesis class of neural networks f:Rd→Rkf: \mathbb{R}^d \to \mathbb{R}^k. For a training dataset containing nn samples across kk classes with njn_j samples in class j∈{1,…,k}j \in \{1, \dots, k\}, let Sj={i:yi=j}S_j = \{i : y_i = j\}. Define the empirical margin of class jj as γj=min⁡i∈Sj(f(xi)yi−max⁡ℓ≠yif(xi)ℓ)\gamma_j = \min_{i \in S_j} (f(x_i)_{y_i} - \max_{\ell \neq y_i} f(x_i)_\ell).

    Let Pj=P(x∣y=j)P_j = P(x \mid y = j) denote the class-conditional distribution, Lj[f]=Pr⁡(x,y)∼Pj[f(x)y<max⁡ℓ≠yf(x)ℓ]L_j[f] = \Pr_{(x, y) \sim P_j}[f(x)_y < \max_{\ell \neq y} f(x)_\ell] be the test 0-1 classification error on class jj, and Lbal[f]=1k∑j=1kLj[f]L_{\text{bal}}[f] = \frac{1}{k} \sum_{j=1}^k L_j[f] be the balanced 0-1 test error.

    With probability at least 1−δ1 - \delta over the training data draw, for all choices of per-class margins γ1,…,γk>0\gamma_1, \dots, \gamma_k > 0 and all f∈Ff \in \mathcal{F}:

    Lj[f]≤L^γj,j[f]+4γjR^j(F)+ϵj(γj)L_j[f] \le \hat{L}_{\gamma_j, j}[f] + \frac{4}{\gamma_j} \hat{\mathcal{R}}_j(\mathcal{F}) + \epsilon_j(\gamma_j)

    where L^γj,j[f]=1nj∑i∈Sj1{max⁡ℓ≠jf(xi)ℓ>f(xi)j−γj}\hat{L}_{\gamma_j, j}[f] = \frac{1}{n_j} \sum_{i \in S_j} \mathbf{1}\{\max_{\ell \neq j} f(x_i)_\ell > f(x_i)_j - \gamma_j\}, R^j(F)\hat{\mathcal{R}}_j(\mathcal{F}) is the empirical Rademacher complexity on class jj scaling as O(C(F)nj)\mathcal{O}\left(\sqrt{\frac{\mathcal{C}(\mathcal{F})}{n_j}}\right) for hypothesis complexity C(F)\mathcal{C}(\mathcal{F}), and ϵj(γj)=O(log⁡log⁡2(2max⁡∣f(x)∣/γj)+log⁡(1/δ)nj)\epsilon_j(\gamma_j) = \mathcal{O}\left(\sqrt{\frac{\log \log_2(2\max |f(x)| / \gamma_j) + \log(1/\delta)}{n_j}}\right).

    Averaging across all classes gives the balanced generalization error bound:

    Lbal[f]≤1k∑j=1k(L^γj,j[f]+4γjC(F)nj+ϵj(γj))L_{\text{bal}}[f] \le \frac{1}{k} \sum_{j=1}^k \left( \hat{L}_{\gamma_j, j}[f] + \frac{4}{\gamma_j} \sqrt{\frac{\mathcal{C}(\mathcal{F})}{n_j}} + \epsilon_j(\gamma_j) \right)

  3. Knowl 3 — Optimal Margin Scaling via Decision Boundary Shift in Binary Classification

    theoretical result

    In binary classification (k=2k = 2) on separable data (where empirical margin error L^γj,j[f]=0\hat{L}_{\gamma_j, j}[f] = 0), suppose a neural network class F={f+b:b∈R2}\mathcal{F} = \{f + b : b \in \mathbb{R}^2\} contains a base model ff achieving total sum of margins γ1′+γ2′=β>0\gamma_1' + \gamma_2' = \beta > 0 for classes with sample sizes n1n_1 and n2n_2.

    By adjusting the output bias vector to b∗=(b1∗,b2∗)b^* = (b_1^*, b_2^*) with b1∗=(γ1∗−γ1′)/2b_1^* = (\gamma_1^* - \gamma_1')/2 and b2∗=−(γ1∗−γ1′)/2b_2^* = -(\gamma_1^* - \gamma_1')/2, the resulting classifier f∗=f+b∗f^* = f + b^* achieves class margins:

    γ1∗=βn21/4n11/4+n21/4=Cn11/4,γ2∗=βn11/4n11/4+n21/4=Cn21/4\gamma_1^* = \beta \frac{n_2^{1/4}}{n_1^{1/4} + n_2^{1/4}} = \frac{C}{n_1^{1/4}}, \qquad \gamma_2^* = \beta \frac{n_1^{1/4}}{n_1^{1/4} + n_2^{1/4}} = \frac{C}{n_2^{1/4}}

    where C=β(n1n2)1/4n11/4+n21/4C = \beta \frac{(n_1 n_2)^{1/4}}{n_1^{1/4} + n_2^{1/4}}.

    These margins directly minimize the balanced generalization bound min⁡γ1+γ2=β(1γ1n1+1γ2n2)\min_{\gamma_1 + \gamma_2 = \beta} \left( \frac{1}{\gamma_1 \sqrt{n_1}} + \frac{1}{\gamma_2 \sqrt{n_2}} \right), establishing that the optimal per-class margin under slow-rate generalization bounds (nj−1/2n_j^{-1/2}) satisfies γj∝nj−1/4\gamma_j \propto n_j^{-1/4}. Under fast-rate generalization bounds (nj−1n_j^{-1}), the optimal trade-off scales as γj∝nj−1/3\gamma_j \propto n_j^{-1/3}.

  4. Knowl 4 — Deferred Re-balancing Optimization Procedure (DRW)

    algorithm

    Deferred Re-balancing Optimization (DRW) is a two-stage training schedule for class-imbalanced datasets. In Stage 1, the model is trained with standard unweighted empirical risk minimization using the LDAM loss until learning rate decay to ensure high-quality feature representation learning. In Stage 2, per-class inverse-frequency re-weighting is introduced with a decayed learning rate to adjust decision boundaries without disrupting learned feature representations.

    Input: Training dataset D={(xi,yi)}i=1nD = \{(x_i, y_i)\}_{i=1}^n with njn_j examples in class j∈{1,…,k}j \in \{1, \dots, k\}, model fθf_\theta, total steps TT, deferred re-balancing step threshold T0T_0, initial learning rate α\alpha, batch size mm
    Output: Trained model parameters θ\theta
    Initialize parameters θ\theta randomly
    for t=1t = 1 to T0−1T_0 - 1 do
        Sample mini-batch B∼DB \sim D with ∣B∣=m|B| = m
        Compute unweighted LDAM loss: L(fθ)←1m∑(x,y)∈BLLDAM((x,y);fθ)\mathcal{L}(f_\theta) \leftarrow \frac{1}{m} \sum_{(x,y) \in B} \mathcal{L}_{\text{LDAM}}((x, y); f_\theta)
        Update parameters: θ←θ−α∇θL(fθ)\theta \leftarrow \theta - \alpha \nabla_\theta \mathcal{L}(f_\theta)
        Optional: decay learning rate α←α/τ\alpha \leftarrow \alpha / \tau if scheduled
    for t=T0t = T_0 to TT do
        Sample mini-batch B∼DB \sim D with ∣B∣=m|B| = m
        Compute re-weighted LDAM loss: L(fθ)←1m∑(x,y)∈Bny−1⋅LLDAM((x,y);fθ)\mathcal{L}(f_\theta) \leftarrow \frac{1}{m} \sum_{(x,y) \in B} n_y^{-1} \cdot \mathcal{L}_{\text{LDAM}}((x, y); f_\theta)
        Compute mini-batch normalizer: Z←∑(x,y)∈Bny−1Z \leftarrow \sum_{(x,y) \in B} n_y^{-1}
        Update parameters with normalized step: θ←θ−α1Z∇θL(fθ)\theta \leftarrow \theta - \alpha \frac{1}{Z} \nabla_\theta \mathcal{L}(f_\theta)
        Optional: decay learning rate α\alpha if scheduled
    return θ\theta

  5. Knowl 5 — LDAM Margin Formulation for Arbitrary Known Test Label Distributions

    equation

    When the test label distribution is known but not uniform, let njn_j denote the training sample size of class jj, and let nj′n_j' denote the sample size or expected proportion of class j∈{1,…,k}j \in \{1, \dots, k\} in the test distribution. The margin offset Δj\Delta_j in the LDAM loss is generalized to:

    Δj∝(nj′nj)1/4\Delta_j \propto \left( \frac{n_j'}{n_j} \right)^{1/4}

    where the constant of proportionality is normalized so that the maximum margin offset across all classes equals a specified constant (e.g., 0.50.5). When the test label distribution is uniform (nj′n_j' is equal for all jj), this reduces to Δj∝nj−1/4\Delta_j \propto n_j^{-1/4}.

  6. Knowl 6 — Feature Representation Quality of ERM versus Early Re-weighting and Re-sampling

    empirical result

    On imbalanced CIFAR-10 with step imbalance (imbalance ratio ρ=100\rho = 100, minority fraction μ=0.5\mu = 0.5, where frequent classes have 100 times more training examples than minority classes), feature representations extracted prior to learning rate annealing were evaluated across three training schemes: Empirical Risk Minimization (ERM), Re-Weighting (RW), and Re-Sampling (RS).

    Linear classifiers were trained on top of the fixed penultimate layer features using a subset of the balanced validation set, and evaluated on the remainder of the validation set:

    • ERM-trained representations achieved the lowest classification error across both frequent classes (errors ≈0.10\approx 0.10 for classes 0-F through 4-F) and minority classes (errors ≈0.18–0.28\approx 0.18\text{--}0.28 for classes 5-M through 9-M).
    • RW-trained representations yielded substantially worse errors across all classes (errors ≈0.12–0.47\approx 0.12\text{--}0.47).
    • RS-trained representations also degraded relative to ERM across all classes (errors ≈0.10–0.37\approx 0.10\text{--}0.37).

    This demonstrates that unweighted ERM optimization in the early training stages builds significantly more expressive internal representations than applying class re-weighting or re-sampling from scratch.

  7. Knowl 7 — Top-1 and Top-5 Validation Errors on iNaturalist 2018

    data/table

    The iNaturalist 2018 dataset is a large-scale real-world species classification benchmark with 437,513 training images across 8,142 classes exhibiting a natural long-tailed distribution, evaluated on a balanced validation split. Models were trained using ResNet-50 for 90 epochs with learning rate decay at epochs 30 and 60. For deferred re-weighting (DRW), re-weighting starts at epoch 60.

    Loss Schedule Top-1 Error (%) Top-5 Error (%)
    ERM SGD 42.86 21.31
    CB Focal SGD 38.88 18.97
    ERM DRW 36.27 16.55
    LDAM SGD 35.42 16.48
    LDAM DRW 32.00 14.82

    LDAM-DRW outperforms standard ERM by 10.86 percentage points in top-1 error (reducing error from 42.86% to 32.00%) and improves over the previous state-of-the-art class-balanced focal loss (CB Focal) by 6.88 percentage points. Both LDAM and DRW independently improve performance over baseline SGD.

  8. Knowl 8 — Validation Errors of ResNet-32 on Imbalanced CIFAR-10 and CIFAR-100

    data/table

    ResNet-32 was evaluated on artificially imbalanced CIFAR-10 and CIFAR-100 datasets under long-tailed imbalance (exponential decay across class sizes) and step imbalance (minority fraction μ=0.5\mu = 0.5 with equal sizes among frequent classes and equal smaller sizes among minority classes) across imbalance ratios ρ=max⁡ini/min⁡ini∈{10,100}\rho = \max_i n_i / \min_i n_i \in \{10, 100\}. Models were trained for 200 epochs with learning rate decays at epochs 160 and 180 (DRW begins at epoch 160).

    Dataset Imbalanced CIFAR-10 Imbalanced CIFAR-100
    Imbalance Type Long-Tailed Step Long-Tailed Step
    Imbalance Ratio (ρ\rho) 100 10 100 10 100 10 100 10
    ERM 29.64 13.61 36.70 17.50 61.68 44.30 61.45 45.37
    Focal 29.62 13.34 36.09 16.36 61.59 44.22 61.43 46.54
    LDAM 26.65 13.04 33.42 15.00 60.40 43.09 60.42 43.73
    CB RS 29.45 13.21 38.14 15.41 66.56 44.94 66.23 46.92
    CB RW 27.63 13.46 38.06 16.20 66.01 42.88 78.69 47.52
    CB Focal 25.43 12.90 39.73 16.54 63.98 42.01 80.24 49.98
    HG-DRS 27.16 14.03 29.93 14.85 – – – –
    LDAM-HG-DRS 24.42 12.72 24.53 12.82 – – – –
    M-DRW 24.94 13.57 27.67 13.17 59.49 43.78 58.91 44.72
    LDAM-DRW 22.97 11.84 23.08 12.19 57.96 41.29 54.64 40.54

    LDAM-DRW attains the lowest top-1 error in all benchmark conditions. Class-independent uniform margin regularizer with DRW (M-DRW) performs worse than LDAM-DRW, confirming the theoretical importance of label-distribution-aware margin scaling.

  9. Knowl 9 — Validation Errors on Imbalanced Tiny ImageNet

    data/table

    Tiny ImageNet consists of 200 classes of 64×6464 \times 64 images, evaluated under long-tailed and step imbalance with imbalance ratios ρ∈{10,100}\rho \in \{10, 100\} using ResNet-18 trained for 120 epochs (learning rate annealed at epoch 90, with DRW re-balancing beginning at epoch 90).

    Imbalance Type Long-Tailed Step
    Imbalance Ratio (ρ\rho) 100 10 100 10
    Loss Schedule Top-1 Top-5 Top-1 Top-5 Top-1 Top-5 Top-1 Top-5
    ERM SGD 66.19 42.63 50.33 26.68 63.82 44.09 50.89 27.06
    CB SM SGD 72.72 52.62 51.58 28.91 74.90 59.14 54.51 33.23
    ERM DRW 64.57 40.79 50.03 26.19 62.36 40.84 49.17 25.91
    LDAM SGD 64.04 40.46 48.08 24.80 62.54 39.27 49.08 24.52
    LDAM DRW 62.53 39.06 47.22 23.84 60.63 38.12 47.43 23.26

    LDAM-DRW consistently outperforms ERM, LDAM-SGD, and Class-Balanced Softmax (CB SM), achieving the lowest top-1 and top-5 error rates across both imbalance settings.

  10. Knowl 10 — Validation Errors on Imbalanced IMDB Binary Sentiment Classification

    data/table

    The IMDB dataset consists of 50,000 movie reviews for binary sentiment classification. An imbalanced training split was constructed by removing 90% of negative reviews, while keeping the validation set balanced. A two-layer bidirectional LSTM was trained with the Adam optimizer.

    Approach Error on Positive Reviews (%) Error on Negative Reviews (%) Mean Error (%)
    ERM 2.86 70.78 36.82
    RS 7.12 45.88 26.50
    RW 5.20 42.12 23.66
    LDAM-DRW 4.91 30.77 17.84

    Under standard ERM, the model suffers a 70.78% error on minority negative reviews. Re-sampling (RS) and re-weighting (RW) reduce negative review error to 45.88% and 42.12% respectively. LDAM-DRW achieves the lowest minority class error (30.77%) and the lowest overall balanced mean error (17.84%).

Coverage note — Qualitative 3D spherical feature visualizations for MNIST (Figure 5) and the supplementary ablation comparing deferred re-weighting with deferred re-sampling across random seeds (Table 5) were omitted as they are auxiliary to the primary findings.

References

  1. 1.Tiny imagenet visual recognition challenge. URL https://tiny-imagenet.herokuapp.com.
  2. 2.Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. arXiv preprint arXiv:1802.05296, 2018.
  3. 3.Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=rJl0r3R9KX.
  4. 4.Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. In Advances in Neural Information Processing Systems, pages 6240–6249, 2017.
  5. 5.Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249–259, 2018.
  6. 6.Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In International Conference on Machine Learning, 2019.
  7. 7.Kaidi Cao, Yu Rong, Cheng Li, Xiaoou Tang, and Chen Change Loy. Pose-robust face recognition via deep residual equivariant mapping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5187–5196, 2018.
  8. 8.Yair Carmon, Yujia Jin, Aaron Sidford, and Kevin Tian. Variance reduction for matrix games. arXiv preprint arXiv:1907.02056, 2019.
  9. 9.Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321–357, 2002.
  10. 10.Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  11. 11.John C Duchi, Tatsunori Hashimoto, and Hongseok Namkoong. Distributionally robust losses against mixture covariate shifts.
  12. 12.Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303–338, 2010.
  13. 13.Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. arXiv preprint arXiv:1712.06541, 2017.
  14. 14.Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.
  15. 15.Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In European Conference on Computer Vision, pages 87–102. Springer, 2016.
  16. 16.Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning, pages 1934–1943, 2018.
  17. 17.Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on Knowledge & Data Engineering, (9):1263–1284, 2008.
  18. 18.Haibo He and Yunqian Ma. Imbalanced learning: foundations, algorithms, and applications. John Wiley & Sons, 2013.
  19. 19.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  20. 20.J Henry Hinnefeld, Peter Cooman, Nat Mammo, and Rupert Deese. Evaluating fairness metrics in the presence of dataset bias. arXiv preprint arXiv:1809.09245, 2018.
  21. 21.Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5375–5384, 2016.
  22. 22.Chen Huang, Yining Li, Change Loy Chen, and Xiaoou Tang. Deep imbalanced learning for face recognition and attribute prediction. IEEE transactions on pattern analysis and machine intelligence, 2019.
  23. 23.Nathalie Japkowicz and Shaju Stephen. The class imbalance problem: A systematic study. Intelligent data analysis, 6(5):429–449, 2002.
  24. 24.Sham M Kakade, Karthik Sridharan, and Ambuj Tewari. On the complexity of linear prediction: Risk bounds, margin bounds, and regularization. In Advances in neural information processing systems, pages 793–800, 2009.
  25. 25.Salman Khan, Munawar Hayat, Syed Waqas Zamir, Jianbing Shen, and Ling Shao. Striking the right balance with uncertainty. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 103–112, 2019.
  26. 26.Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  27. 27.Vladimir Koltchinskii, Dmitry Panchenko, et al. Empirical margin distributions and bounding the generalization error of combined classifiers. The Annals of Statistics, 30(1):1–50, 2002.
  28. 28.Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32–73, 2017.
  29. 29.Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
  30. 30.Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  31. 31.Buyu Li, Yu Liu, and Xiaogang Wang. Gradient harmonized single-stage detector. arXiv preprint arXiv:1811.05181, 2018.
  32. 32.Yaoyong Li, Hugo Zaragoza, Ralf Herbrich, John Shawe-Taylor, and Jaz Kandola. The perceptron algorithm with uneven margins. In ICML, volume 2, pages 379–386, 2002.
  33. 33.Zeju Li, Konstantinos Kamnitsas, and Ben Glocker. Overfitting of neural nets under class imbalance: Analysis and improvements for segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 402–410. Springer, 2019.
  34. 34.Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014.
  35. 35.Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  36. 36.Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International Conference on Machine Learning, pages 3128–3136, 2018.
  37. 37.Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. In ICML, volume 2, page 7, 2016.
  38. 38.Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 212–220, 2017a.
  39. 39.Yu Liu, Hongyang Li, and Xiaogang Wang. Rethinking feature discrimination and polymerization for large-scale recognition. arXiv preprint arXiv:1710.00870, 2017b.
  40. 40.Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2537–2546, 2019.
  41. 41.Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies-volume 1, pages 142–150. Association for Computational Linguistics, 2011.
  42. 42.Michele Merler, Nalini Ratha, Rogerio S Feris, and John R Smith. Diversity in faces. arXiv preprint arXiv:1901.10436, 2019.
  43. 43.Vaishnavh Nagarajan and Zico Kolter. Deterministic PAC-bayesian generalization bounds for deep networks via generalizing noise-resilience. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Hygn2o0qKX.
  44. 44.Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
  45. 45.Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211–252, 2015.
  46. 46.Li Shen, Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In European conference on computer vision, pages 467–482. Springer, 2016.
  47. 47.Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight-net: Learning an explicit mapping for sample weighting. arXiv preprint arXiv:1902.07379, 2019.
  48. 48.Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822–2878, 2018.
  49. 49.Johan AK Suykens and Joos Vandewalle. Least squares support vector machine classifiers. Neural processing letters, 9(3):293–300, 1999.
  50. 50.Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. arXiv preprint arXiv:1503.01817, 2015.
  51. 51.Grant Van Horn and Pietro Perona. The devil is in the tails: Fine-grained classification in the wild. arXiv preprint arXiv:1709.01450, 2017.
  52. 52.Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769–8778, 2018.
  53. 53.Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu. Additive margin softmax for face verification. IEEE Signal Processing Letters, 25(7):926–930, 2018a.
  54. 54.Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputing, 312:135–153, 2018.
  55. 55.Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In Advances in Neural Information Processing Systems, pages 7029–7039, 2017.
  56. 56.Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7278–7286, 2018b.
  57. 57.Colin Wei and Tengyu Ma. Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentation. arXiv e-prints, art. arXiv:1905.03684, May 2019.
  58. 58.Colin Wei and Tengyu Ma. Improved sample complexities for deep networks and robust classification via an all-layer margin. arXiv preprint arXiv:1910.04284, 2019.
  59. 59.Colin Wei, Jason D Lee, Qiang Liu, and Tengyu Ma. On the margin theory of feedforward neural networks. arXiv preprint arXiv:1810.05369, 2018.
  60. 60.Q Zhong, C Li, Y Zhang, H Sun, S Yang, D Xie, and S Pu. Towards good practices for recognition & detection. In CVPR workshops, 2016.
  61. 61.Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Domain adaptation for semantic segmentation via class-balanced self-training. arXiv preprint arXiv:1810.07911, 2018.

Citation

MLA
Cao, K., et al. “Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss”. arXiv, 2019, https://doi.org/10.48550/arxiv.1906.07413.
APA
Cao, K., Wei, C., Gaidon, A., Arechiga, N., & Ma, T. (2019). Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss. arXiv. https://doi.org/10.48550/arxiv.1906.07413
Chicago
Cao, K., C. Wei, A. Gaidon, N. Arechiga, and T. Ma. 2019. “Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss”. Preprint, ArXiv. https://doi.org/10.48550/arxiv.1906.07413.
Harvard
Cao, K. et al. (2019) “Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss”. arXiv. Available at: https://doi.org/10.48550/arxiv.1906.07413.
Vancouver
1. Cao K, Wei C, Gaidon A, Arechiga N, Ma T (2019) Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss. https://doi.org/10.48550/arxiv.1906.07413

BibTeX

@misc{https://doi.org/10.48550/arxiv.1906.07413,
  doi = {10.48550/ARXIV.1906.07413},
  url = {https://arxiv.org/abs/1906.07413},
  author = {Cao, Kaidi and Wei, Colin and Gaidon, Adrien and Arechiga, Nikos and Ma, Tengyu},
  keywords = {Machine Learning (cs.LG), Computer Vision and Pattern Recognition (cs.CV), Machine Learning (stat.ML), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss},
  publisher = {arXiv},
  year = {2019},
  copyright = {arXiv.org perpetual, non-exclusive license}
}
Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors