Back

Smoothed Adaptive Weighting for Imbalanced Semi-Supervised Learning: Improve Reliability Against Unknown Distribution Data

Zhengfeng LaiChao WangHenrry GunawanSen-Ching S. CheungChen-Nee Chuah

article2022ICML60 citations

Proposes a smoothed adaptive weighting framework that dynamically adjusts consistency loss based on per-class learning difficulty, enabling semi-supervised models to handle severely imbalanced data without prior knowledge of the unlabeled distribution.

Listen

Modern artificial intelligence applications often struggle with real-world data where some categories appear far more frequently than others, known as class imbalance. While semi-supervised learning methods lower costs by training models on small amounts of labeled data alongside vast quantities of unlabeled data, they typically rely on the unrealistic assumption that categories are evenly distributed or that unlabeled data follow the exact same distribution as labeled data. When these assumptions fail, standard models develop severe confirmation bias toward majority categories, virtually ignoring rare categories and causing significant performance drops. Prior attempts to address this challenge frequently require heavy computational overhead or rely on unverified assumptions about the unlabeled dataset.

The article designs and evaluates a self-adaptive framework called Smoothed Adaptive Weighting (SAW) to improve the reliability of semi-supervised classification models when data distributions are imbalanced and unknown. Rather than assuming or attempting to directly estimate the true distribution of unlabeled data, the proposed method estimates the learning difficulty of each class dynamically during training and adjusts the model's loss weights in a smoothed manner to avoid unstable training gradients.

The authors conducted extensive experiments across benchmark image datasets—including CIFAR-10, CIFAR-100, and STL-10 under various imbalance ratios—and tested the method against a high-resolution, gigapixel medical pathology dataset where manual annotation is scarce and the underlying class distribution is unknown. The approach evaluated combinations with leading semi-supervised frameworks (FixMatch and ReMixMatch), competing imbalance methods (such as DARP and CReST), and alternative test settings where evaluation distributions were intentionally reversed to stress-test model robustness.

The findings demonstrate substantial performance gains and robustness. First, incorporating smoothed adaptive weighting improved base semi-supervised algorithms by up to 40.5% in geometric mean scores and up to 15% in balanced accuracy under severe imbalance. Second, when the underlying data distributions between labeled and unlabeled sets were completely mismatched, the framework consistently outperformed specialized competitors like DARP by up to 10.9%. Third, under stress-test scenarios with inverted test distributions, competing methods degraded performance while the proposed method retained superior accuracy, achieving up to a 15% margin over DARP. Fourth, in real-world gigapixel pathology tissue segmentation, applying the method to the base model boosted minority class segmentation overlap (Intersection over Union) by over 22% and DICE scores by 15.9%. Finally, the framework introduced minimal computational overhead, adding only about 5% running time compared to the 20% or more required by competing alignment and resampling methods.

These results indicate that organizations can train reliable computer vision models using minimal labeled data without needing prior knowledge of real-world category frequencies. For high-stakes domains such as medical diagnostics, automated inspection, and fraud detection, this approach reduces the financial and operational burden of expert data labeling while mitigating the risk of models systematically failing on critical minority cases.

Organizations deploying semi-supervised learning in production should adopt smoothed adaptive weighting as a lightweight plug-in to their existing training pipelines. Before broad deployment across distinct operational environments, teams should conduct targeted pilot tests to tune baseline smoothing parameters to the specific scale of their data. While the method demonstrates robust empirical success across image classification and segmentation benchmarks, practitioners should remain cautious when working with non-visual data modalities, as testing was bounded to computer vision and pathology imaging domains.

Southern University of Science and TechnologyUniversity of California, DavisUniversity of Kentuckymedical imagesself-adaptivesemi-supervised learningimbalanced semi-supervised learningsmoothed adaptive weightingclass imbalance
ZJUJeffLai/SAW_SSLLai et al (2022).pdf

No sufficiently relevant recommendations were found.

Cover for Smoothed Adaptive Weighting for Imbalanced Semi-Supervised Learning: Improve Reliability Against Unknown Distribution Data

Abstract

Despite recent promising results on semi-supervised learning (SSL), data imbalance, particularly in the unlabeled dataset, could significantly impact the training performance of a SSL algorithm if there is a mismatch between the expected and actual class distributions. The efforts on how to construct a robust SSL framework that can effectively learn from datasets with unknown distributions remain limited. We first investigate the feasibility of adding weights to the consistency loss and then we verify the necessity of smoothed weighting schemes. Based on this study, we propose a self-adaptive algorithm, named Smoothed Adaptive Weighting (SAW). SAW is designed to enhance the robustness of SSL by estimating the learning difficulty of each class and synthesizing the weights in the consistency loss based on such estimation. We show that SAW can complement recent consistency-based SSL algorithms and improve their reliability on various datasets including three standard datasets and one gigapixel medical imaging application without making any assumptions about the distribution of the unlabeled set.

Topics

medical imagesself-adaptivesemi-supervised learningsupervised machine learningsurface acoustic wavesunsupervised learningdistribution functionstransfer learningestimation method

Keywords

imbalanced semi-supervised learningsmoothed adaptive weightingclass imbalanceconsistency regularizationself-training frameworkpseudo-labelingunknown distribution dataconfirmation biasbalanced accuracy

Table of Contents

  • 1. Introduction
  • 2. Related works
  • 3. Methodology
  • 3.1. Preliminaries
  • 3.2. A deeper look at weighting the consistency loss
  • 3.3. SAW: Smoothed adaptive weighting against unknown distribution data
  • 4. Experiments
  • 4.1. CIFAR10-LT
  • 4.2. CIFAR100-LT and STL-10
  • 4.3. Empirical analysis on SAW
  • 4.4. Additional evaluation on a real-world application
  • 5. Discussion
  • Acknowledgements
  • References
  • Appendix
  • A. Training details
  • A.1. Data source and compute resources
  • A.2. Training settings
  • B. Additional Experimental Results
  • B.1. Distribution estimation
  • B.2. Stress-test
  • B.3. Per-class performance
  • B.4. Alternative for smoothed weighting function
  • B.5. Effect of adaptive weighting
  • B.6. Effect of weights on supervised loss
  • B.7. Effect of the parameter β
  • B.8. Additional evaluation on a pathology application

Knowls

  1. Knowl 1 — SAW adapts consistency weights using pseudo-label counts as class-difficulty estimates

    algorithm

    Smoothed Adaptive Weighting (SAW) is an add-on for pseudo-label-based semi-supervised learning methods that use a consistency loss. It does not estimate or align the true unlabeled class distribution. Instead, it treats classes receiving fewer pseudo-labels as harder for the current model and increases their consistency-loss weights. The labeled-data weights are fixed from labeled class counts; the unlabeled-data weights are recomputed during training. SAW applies smoothed weights to the supervised and consistency losses in the reported implementation.

    Input: Labeled examples with class counts m_k, unlabeled examples, C classes, base SSL pseudo-label rule, optimizer, and training duration T
    Initialize model parameters theta; initialize unlabeled consistency weights w_u uniformly; set fixed labeled weights w_l using the labeled class counts
    for each epoch t from 1 to T do
        Train the model on labeled and unlabeled minibatches using the base SSL objective, with w_l in the supervised loss and w_u in the consistency loss
        For each unlabeled example, obtain its class assignment from the base SSL pseudo-label rule
        For each class k, count q_k, the unlabeled examples assigned to class k
        Set n_hat_k = max(q_k, 1) for every class k
        Set N = (1/C) sum_k (m_k + n_hat_k)
        Set beta = (N - 1)/N as the paper's heuristic smoothing parameter
        For each class k, set w_u,k proportional to (1 - beta)/(1 - beta^n_hat_k)
    end for
    Output: Trained model parameters theta

    The count adjustment prevents a class with no predicted examples from producing a singular effective-number weight. The heuristic for β\beta uses an estimated average number of labeled plus pseudo-labeled examples per class; the paper notes that this is a reference value because the true unlabeled class counts are unknown, and that it can be fine-tuned for a new dataset. SAW can be paired with different consistency-based SSL methods and does not require labeled and unlabeled data to have matching class distributions.

  2. Knowl 2 — SAW improves FixMatch and ReMixMatch on long-tailed CIFAR-10 with matched imbalance

    empirical result

    On CIFAR10-LT, the labeled and unlabeled training sets were constructed with the same imbalance ratio, γl=γu∈{50,100,150}\gamma_l=\gamma_u\in\{50,100,150\}, and the held-out test set was balanced. Scores are balanced accuracy (bACC) / geometric mean (GM), in percent; each setting was run for three trials. FixMatch improved from 79.2/77.8 to 84.0/83.6 at γ=50\gamma=50, from 71.5/66.8 to 77.5/76.3 at γ=100\gamma=100, and from 68.4/59.9 to 71.6/69.7 at γ=150\gamma=150 when combined with SAW. ReMixMatch improved from 81.5/80.2 to 86.3/86.1, from 73.8/69.5 to 77.0/76.0, and from 69.9/62.5 to 71.5/68.9 at the same respective imbalance ratios. SAW also exceeded the reported DARP results for both base SSL methods across all three ratios. Combining SAW with supervised long-tail methods provided further gains; for example, ReMixMatch + SAW + cRT reached 87.6/87.4 at γ=50\gamma=50.

  3. Knowl 3 — SAW helps when labeled and unlabeled CIFAR-10 class distributions differ

    empirical result

    On CIFAR10-LT with labeled imbalance γl=100\gamma_l=100, the unlabeled imbalance was varied independently across γu∈{1,50,150}\gamma_u\in\{1,50,150\}, while the held-out test set remained balanced. For FixMatch, bACC/GM (percent) changed from 68.9/42.8 to 83.9/83.3 at γu=1\gamma_u=1, from 73.9/70.5 to 81.5/80.9 at γu=50\gamma_u=50, and from 69.6/62.6 to 76.8/75.4 at γu=150\gamma_u=150 with SAW. Thus SAW improved the base FixMatch results at all three unlabeled distributions, including the balanced-unlabeled case. It did not beat DARP in that balanced case: DARP obtained 85.4/85.0, compared with SAW's 83.9/83.3. For the imbalanced unlabeled cases, SAW exceeded DARP: DARP obtained 77.3/75.5 at γu=50\gamma_u=50 and 72.9/69.5 at γu=150\gamma_u=150. These results test SAW without assuming that the unlabeled distribution matches the labeled distribution.

  4. Knowl 4 — SAW retains performance when the test distribution is reversed

    empirical result

    In a CIFAR10-LT stress test, labeled and unlabeled training data shared an imbalance ratio γl=γu∈{50,100,150}\gamma_l=\gamma_u\in\{50,100,150\}, but the held-out test distribution was reversed relative to the training distribution. For ReMixMatch, SAW achieved bACC/GM scores of 86.3/86.1, 77.0/76.0, and 71.5/68.9 at the three ratios; the corresponding unmodified ReMixMatch bACC scores were 71.0, 54.7, and 41.5. For FixMatch, SAW achieved 78.7/84.2, 64.3/76.4, and 57.5/70.5, compared with baseline bACC scores of 70.5, 51.0, and 38.5. The paper also reports reversed-test experiments with mismatched labeled and unlabeled distributions: in those settings SAW improved FixMatch by up to 15.4 bACC points and 35.1 GM points, and ReMixMatch by up to 14.0 bACC points and 35.4 GM points. Per-class analyses showed gains especially for minority classes, while also improving majority-class precision.

  5. Knowl 5 — SAW transfers to CIFAR-100 and to STL-10 with unknown unlabeled distribution

    empirical result

    On CIFAR100-LT, where labeled and unlabeled training sets had matching imbalance ratios and the test set was balanced, SAW improved both tested SSL methods at γ=10\gamma=10 and γ=20\gamma=20. For example, at γ=20\gamma=20, ReMixMatch rose from 53.5/42.3 to 55.3/46.3 bACC/GM, and FixMatch rose from 54.0/44.4 to 55.7/49.4. On STL-10, only the labeled set was synthetically made long-tailed; all unlabeled examples were used, so their class distribution was unknown. At labeled imbalance γl=20\gamma_l=20, ReMixMatch improved from 60.1/44.9 to 79.2/77.9, and FixMatch from 63.4/52.6 to 71.9/69.0. The reported scores are bACC/GM in percent on balanced held-out test sets. On STL-10, ReMixMatch + SAW also exceeded ReMixMatch + DARP by 8.3 bACC points and 10.9 GM points at γl=20\gamma_l=20.

  6. Knowl 6 — SAW improves minority- and majority-class segmentation on gigapixel pathology images

    empirical result

    The pathology experiment segmented grey matter (GM) and white matter (WM) in gigapixel slides. Only 0.1% of regions from two training slides were labeled; the remaining regions were treated as unlabeled, and evaluation used 10 held-out slides. Compared with FixMatch, FixMatch + SAW increased GM/WM intersection-over-union from 80.7/43.0 to 84.6/65.4 and GM/WM Dice from 89.4/61.3 to 91.0/77.2. WM was the minority class, and the larger gains were on WM. The unlabeled class distribution was not supplied to SAW. DARP and CReST were not compared on this dataset because its unlabeled distribution was unavailable without expert annotation.

  7. Knowl 7 — Smoothed, rather than uniform or strictly inverse-frequency, weights stabilize consistency training

    empirical result

    The paper first studied weighting the consistency loss under the simplifying assumption that labeled and unlabeled data had the same imbalance ratio, using FixMatch on CIFAR10-LT at γ=100\gamma=100. Strict inverse-frequency weights produced unusually large accumulated gradients and unstable training; uniform weights and strict inverse-frequency weights were also the poorest-performing choices in the reported comparison. Smoothed weighting schemes improved performance, supporting the use of moderate rather than extreme class reweighting in the consistency term.

    For CC classes, class count nkn_k, class weight wkw_k, and smoothing parameter 0≤β<10\leq\beta<1, the effective-number scheme uses

    Ek=1−βnk1−β,wk∝1Ek.E_k=\frac{1-\beta^{n_k}}{1-\beta},\qquad w_k\propto\frac{1}{E_k}.

    At β=0\beta=0 this gives uniform weights; as β\beta approaches 1, it approaches inverse class frequency. The paper also evaluated wk∝n/(n+νnk)w_k\propto n/(n+\nu n_k), where nn is the total number of unlabeled examples and ν\nu controls smoothing, and the alternative wk∝(n/nk)αw_k\propto(n/n_k)^\alpha, where α\alpha controls the weighting strength. These alternatives also yielded results comparable to the effective-number weighting in the reported CIFAR10-LT experiments.

  8. Knowl 8 — Adaptive pseudo-label weighting outperforms fixed weights based on known class frequencies

    empirical result

    A CIFAR10-LT ablation compared FixMatch, fixed smoothed weighting (SW) using the known unlabeled distribution, and adaptive SAW, with a balanced test set and matching labeled/unlabeled imbalance ratios. Their bACC scores at γ=50,100,150\gamma=50,100,150 were, respectively: FixMatch, 79.2, 71.5, 68.4; SW, 82.7, 76.2, 70.8; and SAW, 84.0, 77.5, 71.6. Thus SAW exceeded the fixed-weight variant even though SW had access to the true distribution in this controlled experiment. Another ablation found that weighting the consistency loss mattered more than weighting only the supervised loss: at γ=100\gamma=100, supervised-only weighting reached 72.2 bACC, consistency-only weighting 76.3, and the SAW configuration weighting both losses 77.5. The paper cautions that with an optimal smoothing parameter and known unlabeled distribution, fixed SW can slightly outperform SAW; SAW's advantage is adaptive weighting without that distribution information.

  9. Knowl 9 — SAW adds less reported runtime than pseudo-label alignment or iterative resampling

    empirical result

    The authors report that SAW's additional runtime was about 5% of the original consistency-based SSL algorithm, because it adds class weights without resampling or a separate alignment optimization. They report DARP's additional runtime as up to 20% of the original SSL runtime due to pseudo-label alignment. CReST's additional runtime could be an order of magnitude larger than the original SSL runtime because it iteratively resamples training data and reinitializes the classifier.

  10. Knowl 10 — Benchmark construction and training protocol for the reported evaluations

    experimental setup

    The standard image-classification experiments used Wide ResNet-28-2, Adam with learning rate 0.002, exponential moving average with decay 0.999, batch size 64, and three random-seed trials; the appendix specifies 500 training epochs, while the main experimental description reports around 2.5×1052.5\times10^5 training iterations. CIFAR10-LT used 10 classes with largest labeled and unlabeled class counts m1=1500m_1=1500 and n1=3000n_1=3000. For classes ordered from most to least frequent, counts followed mk=m1γl−(k−1)/(C−1)m_k=m_1\gamma_l^{-(k-1)/(C-1)} and nk=n1γu−(k−1)/(C−1)n_k=n_1\gamma_u^{-(k-1)/(C-1)}, where CC is the number of classes and γl,γu\gamma_l,\gamma_u set the labeled and unlabeled imbalance ratios. CIFAR100-LT used m1=150m_1=150 and n1=300n_1=300 with the same construction. For STL-10, the largest labeled class had 450 examples and all unlabeled data were used, leaving the unlabeled distribution unknown. Unless a distribution-shift test was specified, held-out test sets were balanced. Classification was evaluated using bACC and GM.

Coverage note — Ancillary training-curve and confusion-matrix visualizations are not separate knowls because they corroborate the included adaptive-weighting and distribution-shift results rather than add distinct findings.

References

  1. 1.An, J., Ying, L., and Zhu, Y. Why resampling outperforms reweighting for correcting sampling bias with stochastic gradients. In International Conference on Learning Representations, 2020.
  2. 2.Arazo, E., Ortego, D., Albert, P., O’Connor, N. E., and McGuinness, K. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE, 2020.
  3. 3.Aurelio, Y. S., de Almeida, G. M., de Castro, C. L., and Braga, A. P. Learning from imbalanced data sets with weighted cross-entropy function. Neural Processing Letters, 50(2):1937–1949, 2019.
  4. 4.Berthelot, D., Carlini, N., Cubuk, E. D., Kurakin, A., Sohn, K., Zhang, H., and Raffel, C. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. In International Conference on Learning Representations (ICLR), 2020.
  5. 5.Branco, P., Torgo, L., and Ribeiro, R. P. A survey of predictive modeling on imbalanced domains. ACM Computing Surveys (CSUR), 49(2):31, 2016.
  6. 6.Buda, M., Maki, A., and Mazurowski, M. A. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249–259, 2018.
  7. 7.Busa-Fekete, R., Szörényi, B., Dembczynski, K., and Hüllermeier, E. Online F-measure optimization. In Advances in Neural Information Processing Systems (NeurIPS), volume 28, 2015.
  8. 8.Calderon-Ramirez, S., Moemeni, A., Elizondo, D., Colreavy-Donnelly, S., Chavarria-Estrada, L. F., Molina-Cabello, M. A., et al. Correcting data imbalance for semi-supervised covid-19 detection using x-ray chest images. arXiv preprint arXiv:2008.08496, 2020.
  9. 9.Cao, K., Wei, C., Gaidon, A., Arechiga, N., and Ma, T. Learning imbalanced datasets with label-distribution-aware margin loss. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
  10. 10.Chan, L., Hosseini, M. S., Rowsell, C., Plataniotis, K. N., and Damaskinos, S. Histosegnet: Semantic segmentation of histological tissue type in whole slide images. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pp. 10662–10671, 2019.
  11. 11.Chawla, N. V., Bowyer, K. W., Hall, L. O., and Kegelmeyer, W. P. SMOTE: synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 2002.
  12. 12.Chen, W., Jiang, Z., Wang, Z., Cui, K., and Qian, X. Collaborative global-local networks for memory-efficient segmentation of ultra-high resolution images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8924–8933, 2019.
  13. 13.Coates, A., Ng, A., and Lee, H. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics (AISTATS), 2011.
  14. 14.Cui, Y., Jia, M., Lin, T.-Y., Song, Y., and Belongie, S. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  15. 15.Dong, Q., Gong, S., and Zhu, X. Imbalanced deep learning by minority class incremental rectification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.
  16. 16.He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016.
  17. 17.Huang, C., Li, Y., Change Loy, C., and Tang, X. Learning deep representation for imbalanced classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  18. 18.Hyun, M., Jeong, J., and Kwak, N. Class-imbalanced semi-supervised learning. arXiv preprint arXiv:2002.06815, 2020.
  19. 19.Jamal, M. A., Brown, M., Yang, M.-H., Wang, L., and Gong, B. Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7610–7619, 2020.
  20. 20.Japkowicz, N. The class imbalance problem: Significance and strategies. In Proceedings of the International Conference on Artificial Intelligence (ICAI), 2000.
  21. 21.Kang, B., Xie, S., Rohrbach, M., Yan, Z., Gordo, A., Feng, J., and Kalantidis, Y. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2019.
  22. 22.Kang, B., Xie, S., Rohrbach, M., Yan, Z., Gordo, A., Feng, J., and Kalantidis, Y. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations (ICLR), 2020.
  23. 23.Khan, S. H., Hayat, M., Bennamoun, M., Sohel, F. A., and Togneri, R. Cost-sensitive learning of deep feature representations from imbalanced data. IEEE transactions on neural networks and learning systems, 29(8):3573–3587, 2017.
  24. 24.Kim, J., Hur, Y., Park, S., Yang, E., Hwang, S. J., and Shin, J. Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 14567–14579, 2020.
  25. 25.Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
  26. 26.Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, Department of Computer Science, University of Toronto, 2009.
  27. 27.Kubat, M. Addressing the curse of imbalanced training sets: One-sided selection. In Proceedings of the International Conference on Machine Learning (ICML), pp. 179–186, 1997.
  28. 28.Lai, Z., Wang, C., Oliveira, L. C., Dugger, B. N., Cheung, S.-C., and Chuah, C.-N. Joint semi-supervised and active learning for segmentation of gigapixel pathology images with cost-effective labeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 591–600, 2021.
  29. 29.Lee, H., Shin, S., and Kim, H. Abc: Auxiliary balanced classifier for class-imbalanced semi-supervised learning. Advances in Neural Information Processing Systems, 34, 2021.
  30. 30.Liu, J., Sun, Y., Han, C., Dou, Z., and Li, W. Deep representation learning on long-tailed data: A learnable embedding augmentation perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2970–2979, 2020.
  31. 31.Liu, Z., Miao, Z., Zhan, X., Wang, J., Gong, B., and Yu, S. X. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2537–2546, 2019.
  32. 32.Menon, A. K., Jayasumana, S., Rawat, A. S., Jain, H., Veit, A., and Kumar, S. Long-tail learning via logit adjustment. In International Conference on Learning Representations, 2020.
  33. 33.Miyato, T., Maeda, S.-i., Koyama, M., and Ishii, S. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.
  34. 34.Oliver, A., Odena, A., Raffel, C. A., Cubuk, E. D., and Goodfellow, I. Realistic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems (NeurIPS), pp. 3235–3246, 2018.
  35. 35.Oskal, K., Risdal, M., Janssen, E., Undersrud, E., and Gulsrud, T. A U-net based approach to epidermal tissue segmentation in whole slide histopathological images. SN Appl. Sci., 1:672, 06 2019. doi: 10.1007/s42452-019-0694-y.
  36. 36.Pouyanfar, S., Tao, Y., Mohan, A., Tian, H., Kaseb, A. S., Gauen, K., Dailey, R., Aghajanzadeh, S., Lu, Y.-H., Chen, S.-C., et al. Dynamic sampling in convolutional neural networks for imbalanced data classification. In 2018 IEEE conference on multimedia information processing and retrieval (MIPR), pp. 112–117. IEEE, 2018.
  37. 37.Pulido, J. V., Guleria, S., Ehsan, L., Fasullo, M., Lippman, R., Mutha, P., Shah, T., Syed, S., and Brown, D. E. Semi-supervised classification of noisy, gigapixel histology images. In 2020 IEEE 20th International Conference on Bioinformatics and Bioengineering (BIBE), pp. 563–568. IEEE, 2020.
  38. 38.Rahman, M. A. and Wang, Y. Optimizing intersection-over-union in deep neural networks for image segmentation. In International symposium on visual computing, pp. 234–244. Springer, 2016.
  39. 39.Sohn, K., Berthelot, D., Carlini, N., Zhang, Z., Zhang, H., Raffel, C. A., Cubuk, E. D., Kurakin, A., and Li, C.-L. FixMatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
  40. 40.Tai, K. S., Bailis, P., and Valiant, G. Sinkhorn label allocation: Semi-supervised classification via annealed self-training. In International Conference on Machine Learning (ICML), 2021.
  41. 41.Tan, J., Wang, C., Li, B., Li, Q., Ouyang, W., Yin, C., and Yan, J. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11662–11671, 2020.
  42. 42.Tang, Z., Chuang, K. V., DeCarli, C., Jin, L.-W., Beckett, L., Keiser, M. J., and Dugger, B. N. Interpretable classification of alzheimer’s disease pathologies with a convolutional neural network pipeline. Nature communications, 10(1):1–14, 2019.
  43. 43.Tarvainen, A. and Valpola, H. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
  44. 44.Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S. The inaturalist species classification and detection dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  45. 45.Wang, Y.-X., Ramanan, D., and Hebert, M. Learning to model the tail. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
  46. 46.Wei, C., Sohn, K., Mellina, C., Yuille, A., and Yang, F. CReST: A class-rebalancing self-training framework for imbalanced semi-supervised learning. arXiv preprint arXiv:2102.09559, 2021.
  47. 47.Yang, Y. and Xu, Z. Rethinking the value of labels for improving class-imbalanced learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 19290–19301, 2020.
  48. 48.Zhang, B., Wang, Y., Hou, W., Wu, H., Wang, J., Okumura, M., and Shinozaki, T. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 34, 2021.
  49. 49.Zou, K. H., Warfield, S. K., Bharatha, A., Tempany, C. M., Kaus, M. R., Haker, S. J., Wells III, W. M., Jolesz, F. A., and Kikinis, R. Statistical validation of image segmentation quality based on a spatial overlap index1: scientific reports. Academic radiology, 11(2):178–189, 2004.

Citation

MLA
Lai, Z., et al. “Smoothed Adaptive Weighting for Imbalanced Semi-Supervised Learning: Improve Reliability Against Unknown Distribution Data”. International Conference on Machine Learning, vol. 162, 2022, pp. 11828–43, https://proceedings.mlr.press/v162/lai22b.html.
APA
Lai, Z., Wang, C., Gunawan, H., Cheung, S.-C. S., & Chuah, C.-N. (2022). Smoothed Adaptive Weighting for Imbalanced Semi-Supervised Learning: Improve Reliability Against Unknown Distribution Data. International Conference on Machine Learning, 162, 11828–11843. https://proceedings.mlr.press/v162/lai22b.html
Chicago
Lai, Z., C. Wang, H. Gunawan, S.-C. S. Cheung, and C.-N. Chuah. 2022. “Smoothed Adaptive Weighting for Imbalanced Semi-Supervised Learning: Improve Reliability Against Unknown Distribution Data”. International Conference on Machine Learning 162: 11828–43. https://proceedings.mlr.press/v162/lai22b.html.
Harvard
Lai, Z. et al. (2022) “Smoothed Adaptive Weighting for Imbalanced Semi-Supervised Learning: Improve Reliability Against Unknown Distribution Data”, International Conference on Machine Learning. PMLR, pp. 11828–11843. Available at: https://proceedings.mlr.press/v162/lai22b.html.
Vancouver
1. Lai Z, Wang C, Gunawan H, Cheung S-CS, Chuah C-N (2022) Smoothed Adaptive Weighting for Imbalanced Semi-Supervised Learning: Improve Reliability Against Unknown Distribution Data. In: International Conference on Machine Learning. PMLR, pp 11828–11843

BibTeX

@InProceedings{pmlr-v162-lai22b,
  title = 	 {Smoothed Adaptive Weighting for Imbalanced Semi-Supervised Learning: Improve Reliability Against Unknown Distribution Data},
  author =       {Lai, Zhengfeng and Wang, Chao and Gunawan, Henrry and Cheung, Sen-Ching S and Chuah, Chen-Nee},
  booktitle = 	 {Proceedings of the 39th International Conference on Machine Learning},
  pages = 	 {11828--11843},
  year = 	 {2022},
  editor = 	 {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan},
  volume = 	 {162},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {17--23 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v162/lai22b/lai22b.pdf},
  url = 	 {https://proceedings.mlr.press/v162/lai22b.html},
  abstract = 	 {Despite recent promising results on semi-supervised learning (SSL), data imbalance, particularly in the unlabeled dataset, could significantly impact the training performance of a SSL algorithm if there is a mismatch between the expected and actual class distributions. The efforts on how to construct a robust SSL framework that can effectively learn from datasets with unknown distributions remain limited. We first investigate the feasibility of adding weights to the consistency loss and then we verify the necessity of smoothed weighting schemes. Based on this study, we propose a self-adaptive algorithm, named Smoothed Adaptive Weighting (SAW). SAW is designed to enhance the robustness of SSL by estimating the learning difficulty of each class and synthesizing the weights in the consistency loss based on such estimation. We show that SAW can complement recent consistency-based SSL algorithms and improve their reliability on various datasets including three standard datasets and one gigapixel medical imaging application without making any assumptions about the distribution of the unlabeled set.}
}
Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: https://creativecommons.org/licenses/by/4.0/