UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation

Huimin HuangLanfen LinRuofeng TongHongjie HuQiaowei ZhangYutaro IwamotoXianhua HanYen-Wei ChenJian Wu

article2020ICASSP2,799 citations

Introduces UNet 3+, a medical image segmentation architecture that combines full-scale skip connections with deep supervision to capture multi-scale anatomical structures with higher accuracy and fewer parameters than previous UNet variants.

Listen

Medical image segmentation is essential for clinical tasks such as organ analysis in CT scans, yet existing networks like UNet and UNet++ often fail to fully exploit multi-scale features, leading to imprecise boundaries and over-segmentation on non-organ images. These shortcomings matter now because accurate automated segmentation can improve diagnostic efficiency and treatment planning amid growing imaging volumes.

The article set out to develop and evaluate UNet 3+, a redesigned encoder-decoder network that incorporates full-scale skip connections, deep supervision, a hybrid loss function, and a classification-guided module to achieve higher accuracy with fewer parameters.

The approach involved redesigning skip connections to aggregate features across all scales and adding side outputs supervised by a combination of focal, MS-SSIM, and IoU losses, plus a binary classification branch to suppress false positives. Experiments used the ISBI LiTS 2017 liver dataset of 131 CT volumes and a spleen dataset of 49 CT volumes from a hospital, with performance measured by Dice coefficient against UNet, UNet++, and other state-of-the-art methods.

UNet 3+ reduced parameters substantially compared with prior U-Net variants while raising average Dice scores by 2.7 to 3.4 points over UNet and UNet++ across both backbones and datasets. Adding full-scale deep supervision contributed an extra 0.4-point gain, and the hybrid loss plus classification module further lifted liver Dice to 0.9675 and spleen Dice to 0.9620, outperforming the next-best published results by roughly 3 points. Qualitative results showed clearer organ boundaries and fewer false positives even for small or variably scaled structures.

These gains translate to more reliable position-aware and boundary-enhanced maps that can reduce manual correction time and lower risks of missed or erroneous delineations in clinical workflows. The efficiency improvement also supports deployment on standard hardware without sacrificing accuracy.

The findings support integrating UNet 3+ into segmentation pipelines for abdominal CT, with immediate next steps being broader validation across additional organs and modalities plus pilot deployment in clinical settings. Main limitations include evaluation on only two organs and two datasets; readers should confirm generalizability before large-scale adoption.

Ritsumeikan University Sir Run Run Shaw Hospital Zhejiang Lab Zhejiang University classification computational efficiency encoder-decoder UNet 3+Full-Scale Skip Connection Deep Supervision

arXiv: 2004.08790 ZJ

Paper: U-Net: Convolutional Networks for Biomedical Image Segmentation, Olaf Ronneberger et al. (2015). This paper establishes the classic U-Net encoder-decoder architecture with skip connections that UNet 3+ directly builds upon and enhances.
Paper: UNet++: A Nested U-Net Architecture for Medical Image Segmentation, Zongwei Zhou et al. (2018). This work introduces nested and dense skip connections alongside deep supervision, providing the core benchmark and architectural predecessor that UNet 3+ redesigns to full-scale connections.
Paper: UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation, Zongwei Zhou et al. (2019). This paper expands multiscale feature aggregation and pruning in UNet++, detailing the dense skip pathways that UNet 3+ seeks to simplify and improve.
Paper: Attention U-Net: Learning Where to Look for the Pancreas, Ozan Oktay et al. (2018). This study introduces attention mechanisms into U-Net skip pathways to eliminate false positives in organ segmentation, motivating UNet 3+'s classification-guided module.
Paper: V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation, Fausto Milletari et al. (2016). This paper establishes end-to-end volumetric segmentation and the Dice-based objective function foundational to the hybrid loss strategies used in UNet 3+.
Paper: Fully convolutional networks for semantic segmentation, Jonathan Long et al. (2015). This seminal paper introduces fully convolutional architectures and multiscale feature fusion via skip layers, underpinning all modern encoder-decoder segmentation networks.

Paper: Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation, Hu Cao et al. (2021). This paper extends U-shaped medical segmentation by replacing convolutional feature extractors with pure Swin Transformer blocks to capture long-range contextual dependencies.
Paper: UNETR: Transformers for 3D Medical Image Segmentation, Ali Hatamizadeh et al. (2021). This work advances beyond convolutional multiscale skip architectures by formulating 3D medical image segmentation with a vision transformer encoder connected to a CNN decoder.
Paper: Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images, Ali Hatamizadeh et al. (2022). This paper builds on transformer-augmented multiscale decoders by integrating shifted-window hierarchical self-attention for complex volumetric medical segmentation tasks.
Paper: U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation, Chenxin Li et al. (2025). This study pushes the evolution of U-Net architectures further by substituting linear layers with Kolmogorov-Arnold Network blocks to enhance nonlinear representation and parameter efficiency.

Cover for UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation

Abstract

Recently, a growing interest has been seen in deep learning-based semantic segmentation. UNet, which is one of deep learning networks with an encoder-decoder architecture, is widely used in medical image segmentation. Combining multi-scale features is one of important factors for accurate segmentation. UNet++ was developed as a modified Unet by designing an architecture with nested and dense skip connections. However, it does not explore sufficient information from full scales and there is still a large room for improvement. In this paper, we propose a novel UNet 3+, which takes advantage of full-scale skip connections and deep supervisions. The full-scale skip connections incorporate low-level details with high-level semantics from feature maps in different scales; while the deep supervision learns hierarchical representations from the full-scale aggregated feature maps. The proposed method is especially benefiting for organs that appear at varying scales. In addition to accuracy improvements, the proposed UNet 3+ can reduce the network parameters to improve the computation efficiency. We further propose a hybrid loss function and devise a classification-guided module to enhance the organ boundary and reduce the over-segmentation in a non-organ image, yielding more accurate segmentation results. The effectiveness of the proposed method is demonstrated on two datasets. The code is available at: this http URL

1. INTRODUCTION
2. METHODS
2.1. Full-scale Skip Connections
2.2. Full-scale Deep Supervision
2.3. Classification-guided Module (CGM)
3. EXPERIMENTS AND RESULTS
3.1. Datasets and Implementation
3.2. Comparison with UNet and UNet++
3.3. Comparison with the State of the Art
4. CONCLUSIONS
5. ACKNOWLEDGMENTS
6. REFERENCES

Knowls

Knowl 1 — Full-Scale Skip Connections in UNet 3+
model/method
UNet 3+ redesigns the inter-connections between encoder and decoder as well as intra-connections among decoders to aggregate multi-scale features directly at every decoder stage. For an architecture with $N$ total stages (typically $N=5$ ), let $X_{En}^i$ denote the feature map from the $i$ -th encoder down-sampling stage and $X_{De}^i$ denote the feature map from the $i$ -th decoder stage.

Each decoder stage $X_{De}^i$ ( $i \in \{1, \dots, N-1\}$ ) captures fine-grained details and coarse-grained semantics from all $N$ scales:
- From smaller-scale encoder layers $X_{En}^k$ ( $k=1, \dots, i-1$ ), low-level spatial details are extracted using non-overlapping max-pooling $\mathcal{D}(\cdot)$ .
- From the same-scale encoder layer $X_{En}^i$ , features are passed directly.
- From larger-scale decoder layers $X_{De}^k$ ( $k=i+1, \dots, N$ ), high-level semantic information is transmitted using bilinear upsampling $\mathcal{U}(\cdot)$ .
To unify channel dimensions and eliminate redundancy, a convolution operation $\mathcal{C}(\cdot)$ with 64 filters of size $3 \times 3$ is applied to each of the $N$ incoming streams. The resulting $N$ feature maps (each having 64 channels) are concatenated along the channel dimension (yielding $64 \times N$ channels) and passed through a feature aggregation mechanism $\mathcal{H}(\cdot)$ , which consists of a $3 \times 3$ convolution with $64 \times N$ (e.g., 320) filters, Batch Normalization, and a ReLU activation:

$X_{De}^i = \begin{cases} X_{En}^i, & i = N \\[8pt] \mathcal{H}\left( \left[ \left( \mathcal{C}(\mathcal{D}(X_{En}^k)) \right)_{k=1}^{i-1}, \mathcal{C}(X_{En}^i), \left( \mathcal{C}(\mathcal{U}(X_{De}^k)) \right)_{k=i+1}^N \right] \right), & i = 1, \dots, N-1 \end{cases}$

where $[\cdot]$ represents channel concatenation.
Knowl 2 — Decoder Parameter Formulation in UNet 3+ vs UNet and UNet++
equation
Let $i$ denote the stage index, $N$ the total number of stages, $D_k$ the convolution kernel spatial dimension (e.g., 3), and $d(\cdot)$ the depth (channel count) of a node's feature map. Assuming an encoder where $d(X_{En}^i) = 32 \times 2^i$ :

In standard UNet, where decoder channels mirror the encoder ( $d(X_{De}^i) = 32 \times 2^i$ ), the parameter count at decoder stage $i$ is: $P_{U-De}^i = D_k \times D_k \times \left( d(X_{De}^{i+1}) \times d(X_{De}^i) + d(X_{De}^i)^2 \right) + \left( d(X_{En}^i) + d(X_{De}^i) \right) \times d(X_{De}^i)$

In UNet++, due to dense intermediate nodes along skip pathways $X_{De}^{i,j}$ : $P_{U++-De}^i = D_k \times D_k \times \left( d(X_{De}^{i+1}) \times d(X_{De}^i) + d(X_{De}^i)^2 \right) + \left( d(X_{En}^i) + \sum_{j=1}^{i-1} d(X_{De}^{i,j}) + d(X_{De}^i) \right) \times d(X_{De}^i)$

In UNet 3+, because each incoming skip connection from encoder stages $1, \dots, i$ and decoder stages $i+1, \dots, N$ is first reduced to 64 channels, the total parameter count at decoder stage $i$ is: $P_{U3+-De}^i = D_k \times D_k \times \left( \sum_{k=1}^i d(X_{En}^k) + \sum_{k=i+1}^N d(X_{De}^k) \right) \times 64 + d(X_{De}^i)^2$

where $d(X_{De}^i) = 64 \times N$ channels. This channel reduction makes UNet 3+ substantially more parameter-efficient than UNet and UNet++.
Knowl 3 — Classification-Guided Module (CGM)
model/method
The Classification-Guided Module (CGM) prevents false positives (over-segmentation) in 2D medical slices that do not contain the target organ. It performs auxiliary binary image-level classification on the deepest encoder feature map $X_{En}^N$ , which contains the highest-level semantic information.

The feature map $X_{En}^N$ is passed through a sequence of Dropout, $1 \times 1$ convolution, Adaptive Max-Pooling, and Sigmoid activation to produce a 2-dimensional probability vector representing the presence or absence of the target organ in the input slice. During training, this auxiliary head is optimized using Binary Cross-Entropy loss.

At inference and during forward propagation, an $\operatorname{argmax}$ operation converts the 2D probability vector into a binary scalar decision $c \in \{0, 1\}$ ( $0$ for organ absent, $1$ for organ present). Each decoder side-output segmentation map $S_i$ is then multiplied by $c$ :

$S_i^{guided} = c \cdot S_i$

When no target organ is present ( $c = 0$ ), all side-output segmentation predictions are mapped to zero, eliminating background false-positive segmentations.
Knowl 4 — Full-Scale Deep Supervision
model/method
Full-scale deep supervision in UNet 3+ generates side-output segmentation maps directly from each decoder stage $X_{De}^i$ ( $i \in \{1, \dots, N\}$ ) rather than solely at full resolution.

For each decoder stage $i$ , the aggregated feature map $X_{De}^i$ is fed into a side-output projection module consisting of:
1. A $3 \times 3$ convolution layer reducing channels to the number of segmentation classes.
2. Bilinear up-sampling to scale the feature map to the original input resolution.
3. A Sigmoid activation function to output predicted pixel probabilities $S_i$ .
Each side output $S_i$ is independently supervised by the full-resolution ground truth mask using a multi-scale hybrid loss function, forcing intermediate decoder layers to learn hierarchical representations of organ position and boundaries.
Knowl 5 — Hybrid Segmentation Loss Function
equation
The hybrid segmentation loss $\ell_{seg}$ combines three loss terms operating at pixel-, patch-, and map-level hierarchies to capture both large-scale organ shape and fine boundary details:

$\ell_{seg} = \ell_{fl} + \ell_{ms-ssim} + \ell_{iou}$

where:
- $\ell_{fl}$ is the focal loss (pixel-level), which addresses foreground-background class imbalance by down-weighting easy background pixels.
- $\ell_{ms-ssim}$ is the multi-scale structural similarity index loss (patch-level), which assigns higher penalty to fuzzy, ambiguous organ boundary regions.
- $\ell_{iou}$ is the Intersection-over-Union loss (map-level), which penalizes global spatial overlap errors across the entire predicted mask.
Knowl 6 — Patch-Level Multi-Scale Structural Similarity Index (MS-SSIM) Loss
equation
For corresponding $N_p \times N_p$ patches $p = \{p_j : j = 1, \dots, N_p^2\}$ cropped from the predicted probability map $P$ and $g = \{g_j : j = 1, \dots, N_p^2\}$ from the ground truth mask $G$ , the MS-SSIM loss is defined as:

$\ell_{ms-ssim} = 1 - \prod_{m=1}^M \left( \frac{2\mu_p \mu_g + C_1}{\mu_p^2 + \mu_g^2 + C_1} \right)^{\beta_m} \left( \frac{2\sigma_{pg} + C_2}{\sigma_p^2 + \sigma_g^2 + C_2} \right)^{\gamma_m}$

where:
- $M$ is the total number of evaluation scales (set to $M = 5$ ).
- $\mu_p$ and $\mu_g$ are the mean intensities of patches $p$ and $g$ .
- $\sigma_p^2$ and $\sigma_g^2$ are the variances of $p$ and $g$ , and $\sigma_{pg}$ is their covariance.
- $\beta_m$ and $\gamma_m$ define the relative importance of luminance and structural/contrast components at scale $m$ .
- $C_1 = 0.01^2$ and $C_2 = 0.03^2$ are small positive constants preventing division by zero.
Knowl 7 — Experimental Setup on Liver and Spleen CT Datasets
experimental setup
UNet 3+ was evaluated on two organ segmentation tasks using abdominal CT scans:
1. Liver Dataset: ISBI LiTS 2017 Challenge dataset containing 131 contrast-enhanced 3D abdominal CT scans (103 scans for training, 28 for testing).
2. Spleen Dataset: Clinical CT dataset containing 49 volumes (40 for training, 9 for testing).
Data Preprocessing and Training Protocol:
- 2.5D context: Input images are formatted as 3-channel slices comprising the target slice together with its adjacent upper and lower slices.
- Resolution: Slices are cropped to $320 \times 320$ pixels.
- Optimization: Stochastic Gradient Descent (SGD) with default hyperparameters.
- Evaluation Metric: Dice similarity coefficient calculated on single-model inference without post-processing (such as CRF).

Knowl 8 — Comparison of UNet, UNet++, and UNet 3+ across Backbones

data/table

Quantitative comparison of parameter count and Dice segmentation accuracy on liver and spleen CT datasets using VGG-16 and ResNet-101 backbones, optimized with focal loss:

Architecture	Vgg-16			ResNet-101			$Dice_{average}$
Architecture	Params	$Dice_{liver}$	$Dice_{spleen}$	Params	$Dice_{liver}$	$Dice_{spleen}$	$Dice_{average}$
UNet	39.39M	0.9206	0.9023	55.90M	0.9387	0.9332	0.9237
UNet++	47.18M	0.9278	0.9230	63.76M	0.9475	0.9423	0.9352
UNet 3+ w/o DS	26.97M	0.9489	0.9437	43.55M	0.9580	0.9539	0.9511
UNet 3+	26.97M	0.9550	0.9496	43.55M	0.9601	0.9560	0.9552

UNet 3+ achieves higher Dice scores while using significantly fewer parameters (26.97M vs 47.18M for VGG-16; 43.55M vs 63.76M for ResNet-101) compared to UNet++. Adding full-scale deep supervision (DS) provides an average improvement of 0.41 percentage points across datasets without altering the test-time parameter footprint.

Knowl 9 — State-of-the-Art Segmentation Comparison and Component Ablation

data/table

Comparison of ResNet-101-based UNet 3+ variants against five baseline semantic segmentation methods on the ISBI LiTS 2017 liver dataset and the spleen CT dataset, reported in Dice similarity coefficient:

Method	$Dice_{liver}$	$Dice_{spleen}$
PSPNet	0.9242	0.9240
DeepLabV2	0.9021	0.9097
DeepLabV3	0.9217	0.9217
DeepLabV3+	0.9186	0.9290
Attention UNet	0.9341	0.9324
UNet 3+ (focal loss)	0.9601	0.9560
UNet 3+ (Hybrid loss)	0.9643	0.9588
UNet 3+ (Hybrid loss + CGM)	0.9675	0.9620

UNet 3+ with focal loss surpasses the strongest baseline (Attention UNet) by +0.0260 on liver and +0.0236 on spleen. Replacing focal loss with the three-level hybrid loss further increases performance to 0.9643 on liver and 0.9588 on spleen. Adding the Classification-Guided Module (CGM) yields the highest accuracy (0.9675 liver, 0.9620 spleen) by suppressing over-segmentation on non-organ slices.

Coverage note — No substantial contributed material was omitted.

References

1.J. Long, E. Shelhamer and T. Darrell, “Fully Convolutional Networks for Semantic Segmentation,” The IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431-3440, 2015.
2.O. Ronneberger, P. Fischer and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” Medical Image Computing and Computer-Assisted Intervention, pp.234-241, 2015.
3.H.S. Zhao, J.P. Shi, X.J. Qi, X.G. Wang and J.Y. Jia, “Pyramid scene parsing network,” The IEEE Conference on Computer Vision and Pattern Recognition, pp. 2881-2890, 2017.
4.L-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy and A.L. Yuille, “Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE transactions on pattern analysis and machine intelligence, vol.40, no.4, pp: 834-848, 2018.
5.L-C. Chen, G. Papandreou, F. Schroff and H. Adam, “Rethinking atrous convolution for semantic image segmentation”, arXiv preprint arXiv:1706.05587, 2017.
6.L-C. Chen, Y.K. Zhu, G. Papandreou and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” Proceedings of the European Conference on Computer Vision, 2018.
7.Z.W. Zhou, M.M.R. Siddiquee, N. Tajbakhsh and J.M. Liang, “UNet++: A Nested U-Net Architecture for Medical Image Segmentation,” Deep Learning in Medical Image Anylysis and Multimodal Learning for Clinical Decision Support, pp: 3-11, 2018.
8.O.O et al., “Attention u-net: Learning where to look for the pancreas,” Medical Imaging with Deep Learning, 2018.
9.Z. Wang, E.P. Simoncelli and A.C. Bovik, “Multiscale structural similarity for image quality assessment,” The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003.
10.T.-Y. Lin, P. Goyal, R. Girshick, K.M. He and P. Dollar. “Focal loss for dense object detection,” The IEEE international conference on computer vision, pp. 2980-2988, 2017.
11.G. Mattyus, W.J. Luo, and R. Urtasun, “Deep-roadmapper: Extracting road topology from aerial images”, The IEEE international conference on computer vision, 2017.
12.P.-T. Boer et al., “A tutorial on the cross-entropy method,” Annals of Operations Research, vol.134, no.1, pp. 19–67, 2005.

Citation

MLA

Huang, H., et al. “UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation”. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 1055–59, https://doi.org/10.1109/ICASSP40776.2020.9053405.

APA

Huang, H., Lin, L., Tong, R., Hu, H., Zhang, Q., Iwamoto, Y., Han, X., Chen, Y.-W., & Wu, J. (2020). UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1055–1059. https://doi.org/10.1109/ICASSP40776.2020.9053405

Chicago

Huang, H., L. Lin, R. Tong, et al. 2020. “UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation”. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1055–59. https://doi.org/10.1109/ICASSP40776.2020.9053405.

Harvard

Huang, H. et al. (2020) “UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation”, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pp. 1055–1059. Available at: https://doi.org/10.1109/ICASSP40776.2020.9053405.

Vancouver

1. Huang H, Lin L, Tong R, Hu H, Zhang Q, Iwamoto Y, Han X, Chen Y-W, Wu J (2020) UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation. In: ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pp 1055–1059

BibTeX

@inproceedings{Huang_2020, title={UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation}, url={http://dx.doi.org/10.1109/ICASSP40776.2020.9053405}, DOI={10.1109/icassp40776.2020.9053405}, booktitle={ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, publisher={IEEE}, author={Huang, Huimin and Lin, Lanfen and Tong, Ruofeng and Hu, Hongjie and Zhang, Qiaowei and Iwamoto, Yutaro and Han, Xianhua and Chen, Yen-Wei and Wu, Jian}, year={2020}, month=May, pages={1055–1059} }

Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Full-Scale Skip Connections in UNet 3+

Knowl 2 — Decoder Parameter Formulation in UNet 3+ vs UNet and UNet++

Knowl 3 — Classification-Guided Module (CGM)

Knowl 4 — Full-Scale Deep Supervision

Knowl 5 — Hybrid Segmentation Loss Function

Knowl 6 — Patch-Level Multi-Scale Structural Similarity Index (MS-SSIM) Loss