ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data

Foivos I. DiakogiannisFrançois WaldnerPeter CaccettaChen Wu

article2019Isprs Journal of Photogrammetry and Remote Sensing2,041 citations

Introduces ResUNet-a, a deep learning architecture that couples conditioned multi-task learning and atrous pyramid pooling with a modified Generalized Dice loss to tackle severe class imbalance in high-resolution aerial image segmentation.

Listen

Automated semantic segmentation of very high resolution aerial imagery is vital for urban planning, infrastructure monitoring, and land-use management. However, extracting accurate pixel-level categories remains difficult in complex urban environments due to high within-class visual diversity, spectral similarities between distinct surfaces, and intricate object boundaries. The article introduces and evaluates ResUNet-a, a deep learning framework designed to produce highly accurate, robust semantic segmentation from high-resolution aerial data.

The proposed framework integrates a multi-scale neural network architecture with a novel loss function. The network uses an encoder-decoder backbone equipped with residual connections, multi-scale atrous convolutions, and pyramid scene parsing pooling layers. Crucially, the model utilizes conditioned multi-task learning, sequentially predicting class boundaries, distance maps, and color reconstructions to inform and refine the final segmentation mask. Alongside the architecture, the authors introduced the Tanimoto loss with complement to handle severe class imbalances and support both classification and continuous regression tasks. The framework was evaluated on the benchmark ISPRS 2D Potsdam urban aerial imagery dataset across multiple configurations and patch scales.

The analysis produced several key findings. First, the best-performing configuration (ResUNet-a d7v2 cmtsk) achieved state-of-the-art results, recording an average F1 score of 92.9% across object classes and an overall accuracy of 91.5%, outperforming existing leading models. Second, incorporating conditioned multi-tasking dramatically improved boundary definition and accelerated model convergence, requiring roughly one-third the training epochs of baseline architectures to achieve equivalent performance. Third, the Tanimoto loss with complement substantially outperformed standard Dice loss variations, achieving a Matthews Correlation Coefficient of 85.99 compared to 80.72 for standard Dice loss under identical training regimes. Finally, the model achieved these top-tier results when trained entirely from scratch without requiring external pre-trained weights.

These findings demonstrate that conditioned multi-task learning and specialized loss functions reduce training instability, eliminate the need for laborious external pre-training, and provide superior spatial delineation. For operational remote sensing workflows, this translates directly to more reliable spatial intelligence and reduced human post-processing costs. Organizations looking to deploy automated mapping pipelines should adopt conditioned multi-task segmentation frameworks and leverage the Tanimoto loss function for imbalanced spatial datasets. Further development should focus on testing transferability across broader geographical regions and exploring lightweight variants to lower computational demands during training.

CSIRO Agriculture & Food CSIRO’s Data61 University of Western Australia classification convolutional neural networks inference ResUNet-a Segmentation Loss

arXiv: 1904.00592 feevos/resuneta

Paper: Road Extraction by Deep Residual U-Net, Zhengxin Zhang et al. (2017). This paper establishes the foundational Deep Residual U-Net architecture for remote sensing segmentation upon which ResUNet-a directly innovates.
Paper: U-Net: Convolutional Networks for Biomedical Image Segmentation, Olaf Ronneberger et al. (2015). It introduces the classic encoder-decoder U-Net architecture that serves as the core structural backbone for ResUNet-a.
Paper: Pyramid Scene Parsing Network, Hengshuang Zhao et al. (2016). This work introduces the pyramid scene parsing pooling module incorporated into ResUNet-a to capture rich global multi-scale context.
Paper: Generalised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations, C. Sudre et al. (2017). It formulates the Generalized Dice loss for highly imbalanced segmentation tasks, which ResUNet-a builds upon and extends for class-imbalanced aerial imagery.
Paper: DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs, Liang-Chieh Chen et al. (2016). It introduces atrous (dilated) convolutions for dense semantic image segmentation, a core mechanism utilized throughout ResUNet-a to widen receptive fields without resolution loss.
Paper: Fully convolutional networks for semantic segmentation, Jonathan Long et al. (2015). This paper pioneered end-to-end fully convolutional networks for semantic segmentation, laying the conceptual groundwork for modern dense prediction models.
Paper: Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics, Alex Kendall et al. (2017). It provides crucial background on multi-task learning and loss formulation across complementary spatial and semantic scene representations.

Paper: Image Segmentation Using Deep Learning: A Survey, Shervin Minaee et al. (2020). This survey provides a comprehensive synthesis of deep learning segmentation paradigms, contextualizing advanced encoder-decoder and multi-task models.
Paper: UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation, Huimin Huang et al. (2020). This work extends U-Net architectures with full-scale multi-scale feature aggregations and hybrid compound loss functions for complex boundary delineations.
Paper: SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers, Enze Xie et al. (2021). It transitions semantic segmentation architectures beyond convolutional and atrous paradigms by presenting an efficient hierarchical Transformer framework.
Paper: Per-Pixel Classification is Not All You Need for Semantic Segmentation, Bowen Cheng et al. (2021). This work advances beyond standard per-pixel segmentation models like ResUNet-a by introducing a unified mask classification paradigm.
Paper: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers, Sixiao Zheng et al. (2020). It reformulates semantic segmentation from convolutional feature mapping to sequence-to-sequence modeling using pure Transformers.
Paper: U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection, Xuebin Qin et al. (2020). It develops a two-level nested residual U-structure that further explores multi-scale context preservation without reliance on standard pre-trained backbones.

Cover for ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data

Abstract

Scene understanding of high resolution aerial images is of great importance for the task of automated monitoring in various remote sensing applications. Due to the large within-class and small between-class variance in pixel values of objects of interest, this remains a challenging task. In recent years, deep convolutional neural networks have started being used in remote sensing applications and demonstrate state of the art performance for pixel level classification of objects. \textcolor{black}{Here we propose a reliable framework for performant results for the task of semantic segmentation of monotemporal very high resolution aerial images. Our framework consists of a novel deep learning architecture, ResUNet-a, and a novel loss function based on the Dice loss. ResUNet-a uses a UNet encoder/decoder backbone, in combination with residual connections, atrous convolutions, pyramid scene parsing pooling and multi-tasking inference. ResUNet-a infers sequentially the boundary of the objects, the distance transform of the segmentation mask, the segmentation mask and a colored reconstruction of the input. Each of the tasks is conditioned on the inference of the previous ones, thus establishing a conditioned relationship between the various tasks, as this is described through the architecture's computation graph. We analyse the performance of several flavours of the Generalized Dice loss for semantic segmentation, and we introduce a novel variant loss function for semantic segmentation of objects that has excellent convergence properties and behaves well even under the presence of highly imbalanced classes.} The performance of our modeling framework is evaluated on the ISPRS 2D Potsdam dataset. Results show state-of-the-art performance with an average F1 score of 92.9% over all classes for our best model.

Topics

classification convolutional neural networks inference parsing algorithm reconstruction scene understanding encoder-decoder object segmentation

Keywords

ResUNet-a Segmentation Loss Image Semantic Segmentation Layer Ground Truth arXiv Learning Abs

1 Introduction
2 Related Work
3 The ResUNet-a framework
3.1 Architecture
3.1.1 ResUNet-a
3.1.2 Multitasking ResUNet-a
3.2 Loss function
3.2.1 Introducing the Tanimoto loss with complement
3.2.2 Experimental comparison with other Dice loss functions
3.2.3 Tanimoto with complement as a regression loss
3.2.4 Generalization to multiclass imbalanced problems
3.3 Data augmentation
3.4 Inference methodology
4 Data and preprocessing
5 Architecture and Tanimoto loss experimental analysis
5.1 Accuracy assessment
5.2 Architecture ablation study
5.3 Performance evaluation of the proposed loss function
6 Results and discussion
6.1 Design of experiments
6.2 Performance of ResUNet-a on the FoV×\times4 dataset
6.3 Comparison with other modeling frameworks
7 Conclusions
References
A Software implementation and training characteristics
B Boundary and distance transform from segmentation mask
C Inference results

Knowls

Knowl 1 — ResUNet-a Architecture and Building Blocks
model/method
ResUNet-a is a fully convolutional encoder-decoder neural network designed for high-resolution semantic segmentation. The architecture replaces standard convolutional blocks in a U-Net backbone with modified residual building blocks (ResBlock-a), incorporates pyramid scene parsing pooling (PSPPooling), and employs nearest-neighbor upsampling with normalized convolutions.

The constituent modules are structured as follows:
1. Input Layer: The raw input image is first projected through a $1 \times 1$ 2D convolution (Conv2D) to expand the input channels to the base filter dimension (e.g., $f=32$ ) without spatial information loss.
2. ResBlock-a Module: Each residual block takes an input tensor $X_l$ and processes it through up to four parallel branches. Each branch contains a sequence of two units, where each unit consists of a BatchNorm layer, a ReLU activation, and a $3 \times 3$ Conv2D layer with a specific atrous dilation rate $d_k \in \{1, 3, 15, 31\}$ (or subsets thereof). The outputs of all parallel atrous branches are summed and added to the residual skip connection $X_l$ via element-wise addition to produce $X_{l+1}$ . Summation rather than concatenation is used to preserve a constant condition number of the Hessian matrix across network depth.
3. Downsampling: In the encoder, feature map downsampling between residual blocks is performed using a $1 \times 1$ Conv2D with a stride of 2.
4. PSPPooling Module: A pyramid scene parsing pooling layer splits its input tensor along the channel dimension into 4 equal partitions. Max pooling is applied across spatial pooling sub-regions of scale ratios $1/1$ , $1/2$ , $1/4$ , and $1/8$ . The pooled features are then restored to the layer's spatial dimension, concatenated, and projected through a $1 \times 1$ normed convolution (Conv2DN: Conv2D followed by BatchNorm).
5. Upsampling and Feature Combination: In the decoder, spatial resolution is doubled using nearest-neighbor interpolation followed by a $1 \times 1$ Conv2DN layer (to prevent checkerboard deconvolution artifacts). Decoder features and corresponding encoder skip features are fused via a Combine module, which applies a ReLU activation to the first input, concatenates it with the second input along the channel dimension, and applies a $1 \times 1$ Conv2DN projection.
Two principal depth variants are defined: ResUNet-a d6 (comprising 6 encoder residual blocks and an initial filter size of 32, totaling $\sim 52\text{M}$ parameters) and ResUNet-a d7 (comprising 7 encoder residual blocks, totaling $\sim 160\text{M}$ parameters). In ResUNet-a d7v2, a reduced 3-branch PSPPooling module ( $1/1$ , $1/2$ , $1/4$ ) is embedded at the middle bottleneck layer in addition to the penultimate layer.
Knowl 2 — Conditioned Multi-Task Learning Framework
model/method
The conditioned multi-tasking variant of ResUNet-a (cmtsk) replaces the single-output semantic segmentation head with a conditioned multi-branch inference structure that simultaneously solves four complementary tasks derived directly from the input image and ground-truth segmentation masks:
1. Distance Transform Map: A continuous representation where each pixel value is in $[0, 1]$ , defined as the Euclidean distance from the pixel to the nearest object boundary normalized to $[0, 1]$ .
2. Boundary Detection: A binary mask identifying pixel boundaries between semantic classes, extracted via a Canny edge detector followed by morphological cross dilation.
3. Segmentation Mask: The primary target multi-class probability distribution per pixel.
4. HSV Reconstruction: An auxiliary reconstruction of the original input image converted into the HSV color space, used to preserve fine-grained spectral and spatial details through the decoder without allowing trivial identity mapping shortcuts.
In the conditioned architecture, the tasks are computed in a sequential dependency graph:
- The shared encoder-decoder backbone produces high-level feature representations.
- The distance transform map is predicted first (omitting PSPPooling to prevent spatial block artifacts).
- The predicted distance map is concatenated with the output of the final PSPPooling layer to predict the boundary logits.
- Both the predicted distance map and the predicted boundary logits are concatenated with the PSPPooling features to serve as the input for the final semantic segmentation logits.
- The HSV color reconstruction branch is computed directly from the decoder features in parallel.
Knowl 3 — Tanimoto Loss with Complement
equation
For a continuous predicted probability vector $p = \{p_i\}_{i=1}^N \in [0, 1]^N$ and corresponding target label vector $l = \{l_i\}_{i=1}^N \in [0, 1]^N$ over $N$ pixels, the Tanimoto coefficient (also known as the $D_3$ variant of the Generalized Dice coefficient) is defined as:

$T(p, l) = \frac{\sum_{i=1}^N p_i l_i}{\sum_{i=1}^N \left(p_i^2 + l_i^2 - p_i l_i\right)}$

The Tanimoto loss with complement $\tilde{T}(p, l)$ is defined as the average of the Tanimoto coefficient evaluated on the predicted and target values and the Tanimoto coefficient evaluated on their dual complements:

$\tilde{T}(p, l) = \frac{T(p, l) + T(1 - p, 1 - l)}{2}$

where $1 - p = \{1 - p_1, 1 - p_2, \dots, 1 - p_N\}$ and $1 - l = \{1 - l_1, 1 - l_2, \dots, 1 - l_N\}$ .

The complement term $T(1 - p, 1 - l)$ prevents gradient stagnation by providing non-zero gradient flow for indices $i$ where $l_i = 0$ , which are otherwise zeroed out in the standard inner product numerator $\sum_i p_i l_i$ . As a result, the gradient field lines of $\tilde{T}(p, l)$ form straight geodesic paths toward the ground-truth target from any point in the unit hypercube, and the second-order Laplacian operator $\nabla^2 \tilde{T}(p, l)$ attains an extremum exactly at the ground-truth point.
Knowl 4 — Multiclass Weighted Tanimoto Loss with Complement
equation
For multi-class semantic segmentation across $C$ classes and $N$ image pixels, the class-weighted Tanimoto coefficient is defined as:

$T(p, l) = \frac{\sum_{J=1}^C w_J \sum_{i=1}^N p_{iJ} l_{iJ}}{\sum_{J=1}^C w_J \sum_{i=1}^N \left(p_{iJ}^2 + l_{iJ}^2 - p_{iJ} l_{iJ}\right)}$

where:
- $p_{iJ} \in [0, 1]$ is the predicted probability that pixel $i$ belongs to class $J$ .
- $l_{iJ} \in \{0, 1\}$ is the binary one-hot ground-truth indicator for pixel $i$ and class $J$ .
- $w_J$ is the class weight determined by the inverse squared volume of true positives for class $J$ :
$w_J = V_J^{-2} = \left(\sum_{i=1}^N l_{iJ}\right)^{-2}$

The multiclass weighted Tanimoto loss with complement is given by:

$\tilde{T}(p, l) = \frac{T(p, l) + T(1 - p, 1 - l)}{2}$

where $1 - p$ and $1 - l$ represent element-wise complements $(1 - p_{iJ})$ and $(1 - l_{iJ})$ for each pixel $i$ and class $J$ .
Knowl 5 — Unified Continuous and Discrete Regression via Bounded Tanimoto Loss
theoretical result
When a neural network's outputs are constrained to the domain $[0, 1]$ (via sigmoid or softmax activation functions), the Tanimoto loss with complement $\tilde{T}(p, l)$ functions validly as a loss function for continuous target variables $l_i \in [0, 1]$ .

Because the gradient magnitude and directional flow of $\tilde{T}(p, l)$ remain strictly symmetric around any arbitrary continuous ground-truth point $l \in [0, 1]^N$ , the gradient vectors produced by discrete classification tasks (e.g., multi-class segmentation, boundary detection) and continuous regression tasks (e.g., normalized distance transforms, HSV channel reconstruction) share identical magnitude scales.

Consequently, multi-task loss optimization can be performed directly by summing the unweighted component losses:

$L_{\text{total}} = \sum_{k=1}^K L_k$

without requiring empirical task-weighting hyperparameters $\alpha_k$ to balance disparate loss formulations (such as combining cross-entropy with mean squared error).
Knowl 6 — Overlapping Sliding-Window Inference Algorithm
algorithm
To eliminate border inference degradation caused by the reduced receptive field context at patch edges during full-tile aerial image inference, overlapping patch evaluation with spatial reflection padding is performed.
Input: Large input image tile $I$ of spatial dimensions $H \times W$ , trained model $M$ , patch size $P = 256$ , stride $S = 64$ , padding $B = 128$
Output: Aggregated predicted class probability tensor $\bar{P}$ of dimensions $H \times W \times C$
$I_{\text{pad}} = \text{ReflectPad}(I, \text{pad\_width}=B)$
Initialize accumulator tensor $A$ of shape $(H + 2B) \times (W + 2B) \times C$ with zeros
Initialize count tensor $K$ of shape $(H + 2B) \times (W + 2B) \times 1$ with zeros
$H_{\text{pad}}, W_{\text{pad}} = \text{Shape}(I_{\text{pad}})$
for $y = 0$ to $H_{\text{pad}} - P$ step $S$ do
    for $x = 0$ to $W_{\text{pad}} - P$ step $S$ do
         $\text{patch} = I_{\text{pad}}[y : y + P, x : x + P]$
         $P_{\text{pred}} = M(\text{patch})$
         $A[y : y + P, x : x + P] = A[y : y + P, x : x + P] + P_{\text{pred}}$
         $K[y : y + P, x : x + P] = K[y : y + P, x : x + P] + 1$
    end for
end for
$P_{\text{avg}} = A / K$
$\bar{P} = P_{\text{avg}}[B : B + H, B : B + W]$
return $\bar{P}$
Knowl 7 — Geometric Affine Data Augmentation Strategy
model/method
To prevent overfitting and render the model invariant to arbitrary affine spatial transformations in remote sensing imagery, a geometric augmentation pipeline is applied during training to every paired input image patch and ground-truth mask:
1. Random Rotation and Scaling: Each patch is transformed by choosing a random rotation center $(c_x, c_y)$ within the patch, a random rotation angle $\theta \in [0, 2\pi)$ , and a random scaling factor $s$ (zoom in/out).
2. Reflection Padding: Any pixel coordinates left unfilled within the $256 \times 256$ window following the affine transformation are populated using reflect padding across image borders.
3. Random Flips: Independent random horizontal and vertical reflections are applied along the $x$ and $y$ axes.
This enforces exact rotational and reflectional symmetry in the training data, matching the spatial characteristics of urban overhead scenes without altering the internal network architecture.
Knowl 8 — Ablation of ResUNet-a Architectural Modules on Convergence
empirical result
An ablation study on the ISPRS Potsdam full-resolution dataset (FoV×1) evaluating model complexity and training convergence rate (measured by the Matthews Correlation Coefficient, MCC, over all classes under fixed hyperparameters) shows consistent, cumulative improvements from each architectural component:
- Wide ResUNet Baseline: A standard residual U-Net without atrous convolutions requires approximately 120 epochs to attain an MCC score of $\sim 0.85$ .
- Atrous Convolutions (ResUNet + Atrous): Introducing parallel atrous convolutions with dilation rates up to $d=31$ inside each ResBlock-a (with total parameter count held identical to the baseline) nearly doubles the training convergence rate, reaching MCC $\sim 0.85$ in approximately 65 epochs.
- Pyramid Pooling (ResUNet + Atrous + PSP): Adding PSPPooling at the middle bottleneck and decoder output increases model representation capacity and accelerates early convergence, though it introduces training variance.
- Conditioned Multitasking (ResUNet + Atrous + PSP + CMTSK): Incorporating conditioned multi-tasking stabilizes gradient updates, reduces training metric variance, and reaches the $\text{MCC} \approx 0.85$ threshold in approximately 40 epochs (a $3\times$ speedup over the baseline ResUNet).
- Middle Bottleneck PSPPooling: Comparing ResUNet-a d7v1 (without middle PSP) against ResUNet-a d7v2 (with middle PSP) confirms that middle bottleneck PSPPooling substantially accelerates convergence toward optimality despite the added model parameters.
Knowl 9 — Convergence Comparison of Tanimoto Loss with Complement versus Dice Loss
empirical result
Under identical network architectures, training hyperparameters, and inverse-squared volume class-weighting schemes on the ISPRS Potsdam FoV×4 dataset, the choice of loss function leads to significant differences in optimization speed and final convergence:
- Standard Dice Loss (with denominator probability summation): Optimization exhibits gradient stagnation, plateauing at a Matthews Correlation Coefficient (MCC) of $80.72\%$ .
- Standard Tanimoto Loss (without complement): Converges toward higher performance but exhibits noticeable metric variance and slower initial convergence.
- Tanimoto Loss with Complement: Achieves the fastest and most stable convergence, reaching an MCC of $85.99\%$ within the same number of epochs, outperforming the standard Dice loss by $5.27\%$ absolute MCC.

Knowl 10 — Comparative Semantic Segmentation Performance on ISPRS Potsdam 2D Benchmark

data/table

The table below compares the performance of ResUNet-a against published state-of-the-art methods on the test set of the ISPRS 2D Potsdam Semantic Labelling Challenge. Evaluation is conducted on full-resolution orthophotos (ground sampling distance 5 cm) using ground-truth masks with eroded boundaries. All reported metrics are F1 scores per class (%), Average F1 score across the 5 target classes excluding Background (%), and Overall Accuracy (OA, %) including the Background class.

Methods	ImSurface	Building	LowVeg	Tree	Car	Avg. F1	OA
UZ_1	89.3	95.4	81.8	80.5	86.5	86.7	85.8
RIT_L7	91.2	94.6	85.1	85.1	92.8	89.8	88.4
RIT_4	92.6	97.0	86.9	87.4	95.2	91.8	90.3
DST_5	92.5	96.4	86.7	88.8	94.7	91.7	90.3
CAS_Y3	92.2	95.7	87.2	87.6	95.6	91.7	90.1
CASIA2	93.3	97.0	87.7	88.4	96.2	92.5	91.1
DPN_MFFL	92.4	96.4	87.8	88.0	95.7	92.1	90.4
HSN+OI+WBP	91.8	95.7	84.4	79.6	88.3	87.9	89.4
ResUNet-a d6 cmtsk	93.0	97.2	87.5	88.4	96.1	92.4	91.0
ResUNet-a d7v2 cmtsk	93.5	97.2	88.2	89.2	96.4	92.9	91.5

ResUNet-a d7v2 cmtsk achieves the highest overall accuracy ( $91.5\%$ ) and highest average F1 score ( $92.9\%$ ), achieving top-rank performance across all individual classes without relying on pre-trained ImageNet backbone weights (unlike CASIA2, RIT_4, and DST_5, which use pre-trained backbones).

Coverage note — Omitted minor implementation details such as the specific multi-GPU Horovod learning rate finder step curve and the verbatim OpenCV Python script listings from Appendices A and B, as their essential algorithmic roles are fully captured in the knowls.

References

1.A. Vadivel, Shamik Sural, A.K.M., 2005. Human color perception in the hsv space and its application in histogram generation for image retrieval. URL: https://doi.org/10.1117/12.586823, doi:10.1117/12.586823.
2.Abraham, N., Khan, N.M., 2018. A novel focal tversky loss function with improved attention u-net for lesion segmentation. CoRR abs/1810.07842. URL: http://arxiv.org/abs/1810.07842, arXiv:1810.07842.
3.Audebert, N., Le Saux, B., Lefevre, S., 2018. Beyond rgb: Very high resolution urban remote sensing with multimodal deep networks. ISPRS Journal of Photogrammetry and Remote Sensing 140, 20–32.
4.Audebert, N., Le Saux, B., Lefvre, S., 2017. Segment-before-detect: Vehicle detection and classification through semantic segmentation of aerial images. Remote Sensing 9. URL: http://www.mdpi.com/2072-4292/9/4/368, doi:10.3390/rs9040368.
5.Audebert, N., Saux, B.L., Lefevre, S., 2016. Semantic segmentation of earth observation data using multimodal and multi-scale deep networks. CoRR abs/1609.06846. URL: http://arxiv.org/abs/1609.06846, arXiv:1609.06846.
6.Baatz, M., Schape, A., 2000. Multiresolution segmentation: An optimization approach for high quality multi-scale image segmentation (ecognition) , 12–23.
7.Badrinarayanan, V., Kendall, A., Cipolla, R., 2015. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. CoRR abs/1511.00561. URL: http://arxiv.org/abs/1511.00561, arXiv:1511.00561.
8.Bertasius, G., Shi, J., Torresani, L., 2015. Semantic segmentation with boundary neural fields. CoRR abs/1511.02674. URL: http://arxiv.org/abs/1511.02674, arXiv:1511.02674.
9.Blaschke, T., Hay, G.J., Kelly, M., Lang, S., Hofmann, P., Addink, E., Feitosa, R.Q., Van der Meer, F., Van der Werff, H., Van Coillie, F., et al., 2014. Geographic object-based image analysis–towards a new paradigm. ISPRS journal of photogrammetry and remote sensing 87, 180–191.
10.Borgefors, G., 1986. Distance transformations in digital images. Comput. Vision Graph. Image Process. 34, 344–371. URL: http://dx.doi.org/10.1016/S0734-189X(86)80047-0, doi:10.1016/S0734-189X(86)80047-0.
11.Chen, L., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L., 2016. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. CoRR abs/1606.00915. URL: http://arxiv.org/abs/1606.00915, arXiv:1606.00915.
12.Chen, L., Papandreou, G., Schroff, F., Adam, H., 2017. Rethinking atrous convolution for semantic image segmentation. CoRR abs/1706.05587. URL: http://arxiv.org/abs/1706.05587, arXiv:1706.05587.
13.Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., Zhang, Z., 2015. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv:1512.01274 .
14.Cheng, G., Wang, Y., Xu, S., Wang, H., Xiang, S., Pan, C., 2017. Automatic road detection and centerline extraction via cascaded end-to-end convolutional neural network. IEEE Transactions on Geoscience and Remote Sensing 55, 3322–3337.
15.Comaniciu, D., Meer, P., 2002. Mean shift: A robust approach toward feature space analysis. IEEE Transactions on pattern analysis and machine intelligence 24, 603–619.
16.Crum, W.R., Camara, O., Hill, D.L.G., 2006. Generalized overlap measures for evaluation and validation in medical image analysis. IEEE Trans. Med. Imaging 25, 1451–1461. URL: http://dblp.uni-trier.de/db/journals/tmi/tmi25.html#CrumCH06.
17.Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L., 2009. ImageNet: A Large-Scale Hierarchical Image Database, in: CVPR09.
18.Dice, L.R., . Measures of the amount of ecologic association between species. Ecology 26, 297–302. doi:10.2307/1932409.
19.Drozdzal, M., Vorontsov, E., Chartrand, G., Kadoury, S., Pal, C., 2016. The importance of skip connections in biomedical image segmentation. CoRR abs/1608.04117. URL: http://arxiv.org/abs/1608.04117, arXiv:1608.04117.
20.Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A., 2010. The pascal visual object classes (voc) challenge. International Journal of Computer Vision 88, 303–338.
21.Goldblatt, R., Stuhlmacher, M.F., Tellman, B., Clinton, N., Hanson, G., Georgescu, M., Wang, C., Serrano-Candela, F., Khandelwal, A.K., Cheng, W.H., et al., 2018. Using landsat and nighttime lights for supervised pixel-based image classification of urban land cover. Remote Sensing of Environment 205, 253–275.
22.Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., 2014. Generative adversarial nets, in: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (Eds.), Advances in Neural Information Processing Systems 27. Curran Associates, Inc., pp. 2672–2680. URL: http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf.
23.Goyal, P., Dollar, P., Girshick, R.B., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., He, K., 2017. Accurate, large minibatch SGD: training imagenet in 1 hour. CoRR abs/1706.02677. URL: http://arxiv.org/abs/1706.02677, arXiv:1706.02677.
24.Gu, Y., Wang, Y., Li, Y., 2019. A survey on deep learning-driven remote sensing image scene understanding: Scene classification, scene retrieval and scene-guided object detection. Applied Sciences 9. URL: https://www.mdpi.com/2076-3417/9/10/2110, doi:10.3390/app9102110.
25.He, K., Girshick, R.B., Dollar, P., 2018. Rethinking imagenet pre-training. CoRR abs/1811.08883. URL: http://arxiv.org/abs/1811.08883, arXiv:1811.08883.
26.He, K., Gkioxari, G., Dollar, P., Girshick, R.B., 2017. Mask R-CNN. CoRR abs/1703.06870. URL: http://arxiv.org/abs/1703.06870, arXiv:1703.06870.
27.He, K., Zhang, X., Ren, S., Sun, J., 2014. Spatial pyramid pooling in deep convolutional networks for visual recognition. CoRR abs/1406.4729. URL: http://arxiv.org/abs/1406.4729, arXiv:1406.4729.
28.He, K., Zhang, X., Ren, S., Sun, J., 2015. Deep residual learning for image recognition. CoRR abs/1512.03385. URL: http://arxiv.org/abs/1512.03385, arXiv:1512.03385.
29.He, K., Zhang, X., Ren, S., Sun, J., 2016. Identity mappings in deep residual networks. CoRR abs/1603.05027. URL: http://arxiv.org/abs/1603.05027, arXiv:1603.05027.
30.Huang, G., Liu, Z., Weinberger, K.Q., 2016. Densely connected convolutional networks. CoRR abs/1608.06993. URL: http://arxiv.org/abs/1608.06993, arXiv:1608.06993.
31.Ioffe, S., Szegedy, C., 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR abs/1502.03167. URL: http://arxiv.org/abs/1502.03167, arXiv:1502.03167.
32.ISPRS, . International society for photogrammetry and remote sensing (isprs) and bsf swissphoto: Wg3 potsdam overhead data. http://www2.isprs.org/commissions/comm3/wg4/tests.html.
33.Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K., 2015. Spatial transformer networks. CoRR abs/1506.02025. URL: http://arxiv.org/abs/1506.02025, arXiv:1506.02025.
34.Kervadec, H., Bouchtiba, J., Desrosiers, C., ric Granger, Dolz, J., Ayed, I.B., 2018. Boundary loss for highly unbalanced segmentation arXiv:1812.07032.
35.Kingma, D.P., Ba, J., 2014. Adam: A method for stochastic optimization. CoRR abs/1412.6980. URL: http://arxiv.org/abs/1412.6980, arXiv:1412.6980.
36.Lambert, M.J., Waldner, F., Defourny, P., 2016. Cropland mapping over sahelian and sudanian agrosystems: A knowledge-based approach using proba-v time series at 100-m. Remote Sensing 8, 232.
37.Langkvist, M., Kiselev, A., Alirezaie, M., Loutfi, A., 2016. Classification and segmentation of satellite orthoimagery using convolutional neural networks. Remote Sensing 8, 329.
38.LeCun, Y., Boser, B.E., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W.E., Jackel, L.D., 1989. Backpropagation applied to handwritten zip code recognition. Neural Computation 1, 541–551. URL: https://doi.org/10.1162/neco.1989.1.4.541, doi:10.1162/neco.1989.1.4.541.
39.Li, E., Femiani, J., Xu, S., Zhang, X., Wonka, P., 2015. Robust rooftop extraction from visible band images using higher order crf. IEEE Transactions on Geoscience and Remote Sensing 53, 4483–4495.
40.Li, S., Jiao, J., Han, Y., Weissman, T., 2016. Demystifying resnet. CoRR abs/1611.01186. URL: http://arxiv.org/abs/1611.01186, arXiv:1611.01186.
41.Li, X., Shao, G., 2014. Object-based land-cover mapping with high resolution aerial photography at a county scale in midwestern usa. Remote Sensing 6, 11372–11390.
42.Lin, T., Goyal, P., Girshick, R.B., He, K., Dollar, P., 2017. Focal loss for dense object detection. CoRR abs/1708.02002. URL: http://arxiv.org/abs/1708.02002, arXiv:1708.02002.
43.Liu, Y., Fan, B., Wang, L., Bai, J., Xiang, S., Pan, C., 2018. Semantic labeling in very high resolution images via a self-cascaded convolutional neural network. ISPRS Journal of Photogrammetry and Remote Sensing 145, 78–95.
44.Liu, Y., Minh Nguyen, D., Deligiannis, N., Ding, W., Munteanu, A., 2017a. Hourglass-shapenetwork based semantic segmentation for high resolution aerial imagery. Remote Sensing 9. URL: http://www.mdpi.com/2072-4292/9/6/522, doi:10.3390/rs9060522.
45.Liu, Y., Piramanayagam, S., Monteiro, S.T., Saber, E., 2017b. Dense semantic labeling of very-high-resolution aerial imagery and lidar with fully-convolutional neural networks and higher-order crfs, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Honolulu, USA.
46.Long, J., Shelhamer, E., Darrell, T., 2014. Fully convolutional networks for semantic segmentation. CoRR abs/1411.4038. URL: http://arxiv.org/abs/1411.4038, arXiv:1411.4038.
47.Lu, X., Yuan, Y., Zheng, X., 2017. Joint dictionary learning for multispectral change detection. IEEE Trans. Cybernetics 47, 884–897.
48.Ma, L., Liu, Y., Zhang, X., Ye, Y., Yin, G., Johnson, B.A., 2019. Deep learning in remote sensing applications: A meta-analysis and review. ISPRS Journal of Photogrammetry and Remote Sensing 152, 166 – 177. URL: http://www.sciencedirect.com/science/article/pii/S0924271619301108, doi:https://doi.org/10.1016/j.isprsjprs.2019.04.015.
49.Marmanis, D., Schindler, K., Wegner, J.D., Galliani, S., Datcu, M., Stilla, U., 2018. Classification with an edge: Improving semantic image segmentation with boundary detection. ISPRS Journal of Photogrammetry and Remote Sensing 135, 158–172.
50.Marmanis, D., Wegner, J.D., Galliani, S., Schindler, K., Datcu, M., Stilla, U., 2016. Semantic segmentation of aerial images with an ensemble of cnns.
51.Matikainen, L., Karila, K., 2011. Segment-based land cover mapping of a suburban areacomparison of high-resolution remotely sensed datasets using classification trees and test field points. Remote Sensing 3, 1777–1804.
52.Matthews, B., 1975. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA) - Protein Structure 405, 442 – 451. URL: http://www.sciencedirect.com/science/article/pii/0005279575901099, doi:https://doi.org/10.1016/0005-2795(75)90109-9.
53.Milletari, F., Navab, N., Ahmadi, S., 2016. V-net: Fully convolutional neural networks for volumetric medical image segmentation. CoRR abs/1606.04797. URL: http://arxiv.org/abs/1606.04797, arXiv:1606.04797.
54.Myint, S.W., Gober, P., Brazel, A., Grossman-Clarke, S., Weng, Q., 2011. Per-pixel vs. object-based classification of urban land cover extraction using high spatial resolution imagery. Remote sensing of environment 115, 1145–1161.
55.Novikov, A.A., Major, D., Lenis, D., Hladuvka, J., Wimmer, M., Buhler, K., 2017. Fully convolutional architectures for multi-class segmentation in chest radiographs. CoRR abs/1701.08816. URL: http://arxiv.org/abs/1701.08816, arXiv:1701.08816.
56.Odena, A., Dumoulin, V., Olah, C., 2016. Deconvolution and checkerboard artifacts. Distill URL: http://distill.pub/2016/deconv-checkerboard/.
57.Paisitkriangkrai, S., Sherrah, J., Janney, P., van den Hengel, A., 2016. Semantic labeling of aerial and satellite imagery. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 9, 2868–2881.
58.Pan, S.J., Yang, Q., 2010. A survey on transfer learning. IEEE Trans. on Knowl. and Data Eng. 22, 1345–1359. URL: http://dx.doi.org/10.1109/TKDE.2009.191, doi:10.1109/TKDE.2009.191.
59.Pan, X., Gao, L., Marinoni, A., Zhang, B., Yang, F., Gamba, P., 2018a. Semantic labeling of high resolution aerial imagery and lidar data with fine segmentation network. Remote Sensing 10. URL: http://www.mdpi.com/2072-4292/10/5/743, doi:10.3390/rs10050743.
60.Pan, X., Gao, L., Zhang, B., Yang, F., Liao, W., 2018b. High-resolution aerial imagery semantic labeling with dense pyramid network. Sensors 18. URL: http://www.mdpi.com/1424-8220/18/11/3774, doi:10.3390/s18113774.
61.Penatti, O.A., Nogueira, K., dos Santos, J.A., 2015. Do deep features generalize from everyday objects to remote sensing and aerial scenes domains?, in: 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 44–51. URL: doi.ieeecomputersociety.org/10.1109/CVPRW.2015.7301382, doi:10.1109/CVPRW.2015.7301382.
62.Piramanayagam, S., Saber, E., Schwartzkopf, W., Koehler, F.W., 2018. Supervised classification of multisensor remotely sensed images using a deep learning framework. Remote Sensing 10. URL: http://www.mdpi.com/2072-4292/10/9/1429, doi:10.3390/rs10091429.
63.Rawat, W., Wang, Z., 2017. Deep convolutional neural networks for image classification: A comprehensive review. Neural Computation 29, 2352–2449. doi:10.1162/neco_a_00990. pMID: 28599112.
64.Ronneberger, O., Fischer, P., Brox, T., 2015. U-net: Convolutional networks for biomedical image segmentation. CoRR abs/1505.04597. URL: http://arxiv.org/abs/1505.04597, arXiv:1505.04597.
65.Ruder, S., 2017. An overview of multi-task learning in deep neural networks. CoRR abs/1706.05098. URL: http://arxiv.org/abs/1706.05098, arXiv:1706.05098.
66.Sergeev, A., Balso, M.D., 2018. Horovod: fast and easy distributed deep learning in TensorFlow. arXiv preprint arXiv:1802.05799 .
67.Sherrah, J., 2016. Fully convolutional networks for dense semantic labelling of high-resolution aerial imagery. CoRR abs/1606.02585. URL: http://arxiv.org/abs/1606.02585, arXiv:1606.02585.
68.Smith, L.N., 2018. A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay. CoRR abs/1803.09820. URL: http://arxiv.org/abs/1803.09820, arXiv:1803.09820.
69.Sørensen, T., 1948. A method of establishing groups of equal amplitude in plant sociology based on similarity of species and its application to analyses of the vegetation on Danish commons. Biol. Skr. 5, 1–34.
70.Sudre, C.H., Li, W., Vercauteren, T., Ourselin, S., Cardoso, M.J., 2017. Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. CoRR abs/1707.03237. URL: http://arxiv.org/abs/1707.03237, arXiv:1707.03237.
71.Taghanaki, S.A., Abhishek, K., Cohen, J.P., Cohen-Adad, J., Hamarneh, G., 2019. Deep semantic segmentation of natural and medical images: A review arXiv:1910.07655.
72.Vincent, L., Soille, P., 1991. Watersheds in digital spaces: an efficient algorithm based on immersion simulations. IEEE Transactions on Pattern Analysis & Machine Intelligence , 583–598.
73.Volpi, M., Tuia, D., 2017. Dense semantic labeling of subdecimeter resolution images with convolutional neural networks. IEEE Transactions on Geoscience and Remote Sensing 55, 881–893.
74.Waldner, F., Hansen, M.C., Potapov, P.V., Low, F., Newby, T., Ferreira, S., Defourny, P., 2017. National-scale cropland mapping based on spectral-temporal features and outdated land cover information. PloS one 12, e0181911.
75.Wen, D., Huang, X., Liu, H., Liao, W., Zhang, L., 2017. Semantic classification of urban trees using very high resolution satellite imagery. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 10, 1413–1424.
76.Xie, S., Tu, Z., 2015. Holistically-nested edge detection. CoRR abs/1504.06375. URL: http://arxiv.org/abs/1504.06375, arXiv:1504.06375.
77.Xie, S.M., Jean, N., Burke, M., Lobell, D.B., Ermon, S., 2015. Transfer learning from deep features for remote sensing and poverty mapping. CoRR abs/1510.00098. URL: http://arxiv.org/abs/1510.00098, arXiv:1510.00098.
78.Yang, H., Wu, P., Yao, X., Wu, Y., Wang, B., Xu, Y., 2018. Building extraction in very high resolution imagery by dense-attention networks. Remote Sensing 10. URL: http://www.mdpi.com/2072-4292/10/11/1768, doi:10.3390/rs10111768.
79.Zagoruyko, S., Komodakis, N., 2016. Wide residual networks. CoRR abs/1605.07146. URL: http://arxiv.org/abs/1605.07146, arXiv:1605.07146.
80.Zhang, H., Dana, K., Shi, J., Zhang, Z., Wang, X., Tyagi, A., Agrawal, A., 2018. Context encoding for semantic segmentation, in: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
81.Zhang, Q., Seto, K.C., 2011. Mapping urbanization dynamics at regional and global scales using multi-temporal dmsp/ols nighttime light data. Remote Sensing of Environment 115, 2320–2329.
82.Zhang, Z., Liu, Q., Wang, Y., 2017. Road extraction by deep residual u-net. CoRR abs/1711.10684. URL: http://arxiv.org/abs/1711.10684, arXiv:1711.10684.
83.Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J., 2017a. Pyramid scene parsing network, in: CVPR.
84.Zhao, W., Du, S., Wang, Q., Emery, W.J., 2017b. Contextually guided very-high-resolution imagery classification with semantic segments. ISPRS Journal of Photogrammetry and Remote Sensing 132, 48 – 60. URL: http://www.sciencedirect.com/science/article/pii/S0924271617300709, doi:https://doi.org/10.1016/j.isprsjprs.2017.08.011.
85.Zhu, J., Park, T., Isola, P., Efros, A.A., 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. CoRR abs/1703.10593. URL: http://arxiv.org/abs/1703.10593, arXiv:1703.10593.
86.Zhu, X.X., Tuia, D., Mou, L., Xia, G., Zhang, L., Xu, F., Fraundorfer, F., 2017. Deep learning in remote sensing: A comprehensive review and list of resources. IEEE Geoscience and Remote Sensing Magazine 5, 8–36. doi:10.1109/MGRS.2017.2762307.

Citation

MLA

Diakogiannis, F. I., et al. “ResUNet-a: A Deep Learning Framework for Semantic Segmentation of Remotely Sensed Data”. ISPRS Journal of Photogrammetry and Remote Sensing, vol. 162, 2020, pp. 94–114, https://doi.org/10.1016/j.isprsjprs.2020.01.013.

APA

Diakogiannis, F. I., Waldner, F., Caccetta, P., & Wu, C. (2020). ResUNet-a: A deep learning framework for semantic segmentation of remotely sensed data. ISPRS Journal of Photogrammetry and Remote Sensing, 162, 94–114. https://doi.org/10.1016/j.isprsjprs.2020.01.013

Chicago

Diakogiannis, F. I., F. Waldner, P. Caccetta, and C. Wu. 2020. “ResUNet-a: A Deep Learning Framework for Semantic Segmentation of Remotely Sensed Data”. ISPRS Journal of Photogrammetry and Remote Sensing 162: 94–114. https://doi.org/10.1016/j.isprsjprs.2020.01.013.

Harvard

Diakogiannis, F.I. et al. (2020) “ResUNet-a: A deep learning framework for semantic segmentation of remotely sensed data”, ISPRS Journal of Photogrammetry and Remote Sensing, 162, pp. 94–114. Available at: https://doi.org/10.1016/j.isprsjprs.2020.01.013.

Vancouver

1. Diakogiannis FI, Waldner F, Caccetta P, Wu C (2020) ResUNet-a: A deep learning framework for semantic segmentation of remotely sensed data. ISPRS Journal of Photogrammetry and Remote Sensing 162:94–114

BibTeX

@article{Diakogiannis_2020, title={ResUNet-a: A deep learning framework for semantic segmentation of remotely sensed data}, volume={162}, ISSN={0924-2716}, url={http://dx.doi.org/10.1016/j.isprsjprs.2020.01.013}, DOI={10.1016/j.isprsjprs.2020.01.013}, journal={ISPRS Journal of Photogrammetry and Remote Sensing}, publisher={Elsevier BV}, author={Diakogiannis, Foivos I. and Waldner, François and Caccetta, Peter and Wu, Chen}, year={2020}, month=Apr, pages={94–114} }

Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — ResUNet-a Architecture and Building Blocks

Knowl 2 — Conditioned Multi-Task Learning Framework

Knowl 3 — Tanimoto Loss with Complement

Knowl 4 — Multiclass Weighted Tanimoto Loss with Complement

Knowl 5 — Unified Continuous and Discrete Regression via Bounded Tanimoto Loss

Knowl 6 — Overlapping Sliding-Window Inference Algorithm

Knowl 7 — Geometric Affine Data Augmentation Strategy

Knowl 8 — Ablation of ResUNet-a Architectural Modules on Convergence