Attention U-Net: Learning Where to Look for the Pancreas

Ozan OktayJo SchlemperLoic Le FolgocMatthew LeeMattias HeinrichKazunari MisawaKensaku MoriSteven McDonaghNils Y HammerlaBernhard Kainz

article2018MIDL7,973 citations

Proposes attention gates for the U-Net architecture that automatically focus on target structures of varying shapes and sizes while suppressing irrelevant background regions, eliminating the need for multi-stage localization pipelines in medical image segmentation.

Listen

The paper introduces a new attention mechanism for convolutional neural networks that enables automatic focus on target structures during medical image segmentation. Manual labeling of CT scans remains slow and inconsistent, and standard models such as U-Net require separate localization stages or cascaded networks when organs vary widely in shape and size, as occurs with the pancreas. These extra stages increase computation and parameter counts without improving feature reuse.

The work set out to demonstrate that soft, learnable attention gates can be inserted directly into a single U-Net architecture to suppress irrelevant background activations and highlight salient regions, thereby removing the need for explicit external localization modules. The authors integrated the gates into the skip connections of a 3D U-Net and trained the resulting Attention U-Net end-to-end on two abdominal CT collections: 150 scans acquired at one center and the public TCIA Pancreas-CT set of 82 scans. Both multi-class and single-organ experiments were performed across varying training-set sizes, with direct comparison to baseline U-Net models of equal and greater capacity.

The attention gates raised pancreas Dice scores by 2–3 percentage points on the larger training split (0.840 versus 0.814) and produced statistically significant gains in recall and surface distance. The same pattern held when training data were reduced to 30 scans and when models were tested on an external dataset. Adding the gates increased parameter count by only 8 percent and inference time by roughly 7 percent, yet outperformed higher-capacity plain U-Nets. On the public benchmark the single-model Attention U-Net reached 81.5 percent Dice after cross-validation, matching or exceeding prior cascaded approaches while using far fewer resources. Attention maps visualized during training showed progressive refinement from coarse organ outlines to fine boundaries.

These results indicate that attention can replace the redundant computation of multi-stage pipelines, lowering both training cost and deployment complexity while maintaining or improving accuracy on small, variable structures. The approach therefore supports faster clinical workflows and broader adoption of automated volumetric measurements.

Further gains are likely from pre-training the backbone network, testing residual connections around the gates, and training larger-resolution 3D models once GPU memory permits. The main limitations are the use of downsampled volumes and the absence of post-processing; performance on other organs or modalities remains to be quantified. Overall, the reported improvements are consistent across datasets and statistically supported, providing a solid basis for replacing cascaded localization-segmentation pipelines in similar tasks.

Aichi Cancer Center Babylon Health HeartFlow Imperial College London Nagoya University University of Lübeck computational efficiency convolutional neural networks image segmentation Attention Gate U‑Net Pancreas Segmentation

arXiv: 1804.03999 ozan-oktay/Attention-Gated-Networks

Paper: U-Net: Convolutional Networks for Biomedical Image Segmentation, Olaf Ronneberger et al. (2015). Reading the original U-Net architecture paper first is essential because the source paper builds directly upon its encoder-decoder structure and skip connections.
Paper: CBAM: Convolutional Block Attention Module, Sanghyun Woo et al. (2018). Understanding convolutional block attention modules provides the foundational knowledge for spatial and channel feature refinement assumed by the source's attention gates.
Paper: Squeeze-and-Excitation Networks, Jie Hu et al. (2018). Squeeze-and-Excitation networks establish the baseline concept of adaptively recalibrating intermediate feature maps that the source adapts for medical image segmentation.

Paper: UNet++: A Nested U-Net Architecture for Medical Image Segmentation, Zongwei Zhou et al. (2018). UNet++ extends the U-Net family by introducing nested dense skip connections and deep supervision, building directly upon architectures like Attention U-Net.
Paper: U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation, Chenxin Li et al. (2025). U-KAN continues the line of medical segmentation research initiated by U-Net and Attention U-Net by substituting Kolmogorov–Arnold networks into the backbone.

Cover for Attention U-Net: Learning Where to Look for the Pancreas

Abstract

We propose a novel attention gate (AG) model for medical imaging that automatically learns to focus on target structures of varying shapes and sizes. Models trained with AGs implicitly learn to suppress irrelevant regions in an input image while highlighting salient features useful for a specific task. This enables us to eliminate the necessity of using explicit external tissue/organ localisation modules of cascaded convolutional neural networks (CNNs). AGs can be easily integrated into standard CNN architectures such as the U-Net model with minimal computational overhead while increasing the model sensitivity and prediction accuracy. The proposed Attention U-Net architecture is evaluated on two large CT abdominal datasets for multi-class image segmentation. Experimental results show that AGs consistently improve the prediction performance of U-Net across different datasets and training sizes while preserving computational efficiency. The code for the proposed architecture is publicly available.

Topics

computational efficiency convolutional neural networks image segmentation medical images proposed architectures architecture designs communication overheads

Keywords

Attention Gate U‑Net Pancreas Segmentation CT Abdomen Soft Attention Grid‑Based Gating Multi‑Class Segmentation Dice Score Computational Efficiency Deep Supervision

1 Introduction
1.1 Related Work
1.2 Contributions
2 Methodology
3 Experiments and Results
4 Discussion and Conclusion

Knowls

Knowl 1 — Additive Attention Gate for Dense Feature Filtering
model/method
Attention Gates (AGs) filter feature responses propagating through CNN skip connections by suppressing activations in task-irrelevant background regions and emphasizing salient foreground structures without requiring explicit region-of-interest (ROI) cropping.

Given an input feature tensor $x^l \in \mathbb{R}^{F_l \times H_x \times W_x \times D_x}$ at layer $l$ (with spatial index $i$ and channel index $c$ ) and a gating signal vector $g \in \mathbb{R}^{F_g \times H_g \times W_g \times D_g}$ extracted from a coarser scale ( $H_g \le H_x, W_g \le W_x, D_g \le D_x$ ), the grid-based additive attention gate computes attention coefficients $\alpha_i^l \in [0, 1]$ according to:

$q_{\text{att}}^l = \psi^T \left(\sigma_1\left(W_x^T x_i^l + W_g^T g_i + b_g\right)\right) + b_\psi$

$\alpha_i^l = \sigma_2\left(q_{\text{att}}^l(x_i^l, g_i; \Theta_{\text{att}})\right)$

$\hat{x}_{i,c}^l = x_{i,c}^l \cdot \alpha_i^l$

where:
- $\sigma_1(z) = \max(0, z)$ is the Rectified Linear Unit (ReLU) activation function.
- $\sigma_2(z) = \frac{1}{1 + \exp(-z)}$ is the sigmoid activation function, chosen over softmax to prevent overly sparse output activations and improve optimization stability.
- $W_x \in \mathbb{R}^{F_l \times F_{\text{int}}}$ and $W_g \in \mathbb{R}^{F_g \times F_{\text{int}}}$ are linear transformations implemented via channel-wise $1 \times 1 \times 1$ convolutions that project $x^l$ and $g$ into an intermediate channel space of dimension $F_{\text{int}}$ .
- $\psi \in \mathbb{R}^{F_{\text{int}} \times 1}$ is a linear mapping (implemented via a $1 \times 1 \times 1$ convolution), with scalar bias $b_\psi \in \mathbb{R}$ and intermediate bias $b_g \in \mathbb{R}^{F_{\text{int}}}$ .
- $\Theta_{\text{att}} = \{W_x, W_g, \psi, b_g, b_\psi\}$ is the set of learnable attention gate parameters.
- When $x^l$ and $g$ have different spatial resolutions, $x^l$ is downsampled to the spatial dimensions of $g$ prior to linear projection, and the resulting attention map $\alpha^l$ is upsampled to $(H_x, W_x, D_x)$ via trilinear interpolation before element-wise multiplication $\hat{x}_{i,c}^l = x_{i,c}^l \cdot \alpha_i^l$ .
- For multi-class segmentation, multi-dimensional attention coefficients can be computed where $\alpha_i^l$ is a vector per spatial location, allowing individual attention channels to specialize on distinct target structures.
Knowl 2 — Attention U-Net Architecture for Volumetric Medical Segmentation
model/method
The Attention U-Net integrates additive attention gates into the standard encoder-decoder U-Net architecture to filter features passed along skip connections before channel concatenation.

Key architectural components include:
1. Encoder-Decoder Backbone: The encoder progressively extracts hierarchical representations by downsampling the 3D volume by a factor of 2 at each scale using max-pooling. Feature extraction at each level is performed by repeated blocks of $3 \times 3 \times 3$ convolutions followed by ReLU activations. The decoder mirrors this path with upsampling layers (by a factor of 2) followed by convolution blocks.
2. Gating on Skip Connections: For skip connections at intermediate scales, the coarse-resolution feature map from the deeper decoder stage acts as the gating query signal $g$ , while the encoder feature map at the corresponding resolution serves as the input $x^l$ . The attention gate computes a spatial attention grid $\alpha^l$ , scales $x^l$ , and passes the gated features $\hat{x}^l$ to be concatenated with the upsampled decoder features.
3. Lowest-Level Skip Connection Exception: Gating is omitted on the shallowest (highest spatial resolution) skip connection, as low-level features do not yet represent data in a sufficiently high-dimensional semantic space.
4. Deep Supervision: Auxiliary loss heads are attached to intermediate decoder stages to force intermediate representations to be semantically discriminative across all spatial scales. This prevents dense final predictions from being reconstructed solely from a small subset of skip connections.
Knowl 3 — Gradient Backpropagation Formulation for Attention Gated Convolutional Layers
equation
During the backward pass of training, attention gates (AGs) modulate gradient propagation, scaling backpropagated errors according to attention weights and down-weighting gradient updates originating from background regions.

For an attention-gated feature activation $\hat{x}_i^l = \alpha_i^l \cdot f(x_i^{l-1}; \Phi^{l-1})$ in layer $l$ , where $\Phi^{l-1}$ denotes the trainable convolution parameters of layer $l-1$ , the parameter gradient update is given by:

$\frac{\partial \hat{x}_i^l}{\partial \Phi^{l-1}} = \alpha_i^l \frac{\partial f(x_i^{l-1}; \Phi^{l-1})}{\partial \Phi^{l-1}} + \frac{\partial \alpha_i^l}{\partial \Phi^{l-1}} x_i^l$

where:
- $\hat{x}_i^l$ is the gated output feature at spatial location $i$ in layer $l$ .
- $x_i^l = f(x_i^{l-1}; \Phi^{l-1})$ is the ungated feature output produced by layer $l-1$ .
- $\alpha_i^l \in [0, 1]$ (or an attention vector in multi-dimensional AGs) is the attention gating coefficient computed at spatial location $i$ .
- $\Phi^{l-1}$ is the set of trainable convolutional kernel parameters in layer $l-1$ .
The first term on the right-hand side is directly scaled by $\alpha_i^l$ . Consequently, parameters in shallower layers are updated predominantly based on gradients originating from task-relevant foreground regions where $\alpha_i^l \approx 1$ , while updates from background regions where $\alpha_i^l \approx 0$ are suppressed.
Knowl 4 — Training and Implementation Setup for 3D Attention U-Net
experimental setup
The 3D Attention U-Net segmentation framework is trained under the following experimental protocol:
- Data Preprocessing: CT volumes are resampled to an isotropic voxel resolution of $2.00\text{ mm}$ to accommodate 3D GPU memory constraints. Voxel intensities are linearly normalized to zero mean and unit variance, $\mathcal{N}(0, 1)$ .
- Loss Function: Multi-class Sørensen-Dice loss computed over all semantic target classes simultaneously: $\mathcal{L}_{\text{Dice}} = 1 - \frac{2 \sum_{c} \sum_{i} p_{i,c} y_{i,c} + \epsilon}{\sum_{c} \sum_{i} (p_{i,c}^2 + y_{i,c}^2) + \epsilon}$ where $p_{i,c}$ is the predicted probability and $y_{i,c}$ is the ground truth binary indicator for voxel $i$ and class $c$ .
- Optimization: Adam optimizer with mini-batch sizes of 2 to 4 samples. For networks with high memory footprints, gradients are averaged across multiple forward and backward accumulations before parameter updates.
- Regularization and Augmentation: Batch normalization, deep supervision at intermediate scales, and data augmentation including random affine transformations, axial flips, and random 3D crops.
- Gate Initialization: Gating parameters $\Theta_{\text{att}}$ are initialized such that attention coefficients $\alpha$ initially equal 1 (uniform pass-through) across all spatial coordinates, allowing end-to-end training from scratch without multi-stage training heuristics or reinforcement learning sampling.

Knowl 5 — Multi-Class Abdominal CT Segmentation Performance on the CT-150 Dataset

data/table

Performance of 3D U-Net and 3D Attention U-Net evaluated on the CT-150 dataset (150 contrast-enhanced abdominal 3D CT scans with annotations for pancreas, spleen, and kidney) under two split configurations: 120 training / 30 testing scans, and a reduced-data split of 30 training / 120 testing scans. Evaluation metrics include Dice Similarity Coefficient (DSC), Precision, Recall, and mesh surface-to-surface distance (S2S) in millimeters (reported for pancreas). Inference time is measured for an input tensor size of $160 \times 160 \times 96$ .

Method (Train/Test Split)	U-Net (120/30)	Att U-Net (120/30)	U-Net (30/120)	Att U-Net (30/120)
Pancreas DSC	$0.814 \pm 0.116$	$\mathbf{0.840 \pm 0.087}$	$0.741 \pm 0.137$	$\mathbf{0.767 \pm 0.132}$
Pancreas Precision	$0.848 \pm 0.110$	$0.849 \pm 0.098$	$0.789 \pm 0.176$	$0.794 \pm 0.150$
Pancreas Recall	$0.806 \pm 0.126$	$\mathbf{0.841 \pm 0.092}$	$0.743 \pm 0.179$	$\mathbf{0.762 \pm 0.145}$
Pancreas S2S Dist (mm)	$2.358 \pm 1.464$	$\mathbf{1.920 \pm 1.284}$	$3.765 \pm 3.452$	$3.507 \pm 3.814$
Spleen DSC	$0.962 \pm 0.013$	$0.965 \pm 0.013$	$0.935 \pm 0.095$	$\mathbf{0.943 \pm 0.092}$
Kidney DSC	$0.963 \pm 0.013$	$0.964 \pm 0.016$	$0.951 \pm 0.019$	$0.954 \pm 0.021$
Number of Params	5.88 M	6.40 M	5.88 M	6.40 M
Inference Time	0.167 s	0.179 s	0.167 s	0.179 s

The addition of attention gates yields statistically significant improvements in pancreas DSC (from 0.814 to 0.840 for 120 training cases, $p = .005$ ; and from 0.741 to 0.767 for 30 training cases, $p = .01$ ) and lowers the mean surface distance from $2.358\text{ mm}$ to $1.920\text{ mm}$ . The primary driver of this improvement is an increase in pancreas recall (from 0.806 to 0.841) with minimal computational overhead ( $+0.52\text{ M}$ parameters, $+12\text{ ms}$ inference time).

Knowl 6 — Segmentation Efficiency Comparison: Attention Gates vs Uniform Capacity Increase

data/table

To test whether the performance gains of Attention U-Net stem simply from increased parameter capacity or from the attention gating mechanism itself, standard U-Net models were trained with increased numbers of convolutional filters distributed uniformly across all layers on the CT-150 dataset (120 training / 30 testing scans).

Method	Panc. DSC	Panc. Precision	Panc. Recall	S2S Dist (mm)	# of Pars	Run Time
U-Net (Standard)	$0.814 \pm 0.116$	$0.848 \pm 0.110$	$0.806 \pm 0.126$	$2.358 \pm 1.464$	5.88 M	0.167 s
Attention U-Net	$\mathbf{0.840 \pm 0.087}$	$0.849 \pm 0.098$	$\mathbf{0.841 \pm 0.092}$	$\mathbf{1.920 \pm 1.284}$	6.40 M	0.179 s
U-Net (Capacity+)	$0.821 \pm 0.119$	$0.849 \pm 0.111$	$0.814 \pm 0.125$	$2.383 \pm 1.918$	6.44 M	0.191 s
U-Net (Capacity++)	$0.825 \pm 0.104$	$0.861 \pm 0.082$	$0.807 \pm 0.121$	$2.202 \pm 1.144$	10.40 M	0.222 s

The results demonstrate that simply scaling baseline U-Net capacity uniformly across all layers to 6.44M parameters (0.821 DSC) or 10.40M parameters (0.825 DSC) yields significantly lower segmentation accuracy ( $p = .007$ ) than dedicating parameters specifically to attention gates in Attention U-Net (6.40M parameters, 0.840 DSC), while also requiring lower inference runtime (0.179 s vs 0.191 s and 0.222 s).

Knowl 7 — Pancreas Segmentation Performance on TCIA Pancreas-CT Under Multiple Training Regimes

data/table

Evaluation on the publicly available NIH-TCIA Pancreas-CT benchmark dataset (82 contrast-enhanced abdominal 3D CT scans, split into 61 training and 21 testing scans) across three training settings:

Before Fine-Tuning (BFT): Models trained exclusively on the CT-150 dataset and directly evaluated on TCIA Pancreas-CT without adaptation.
After Fine-Tuning (AFT): Models pre-trained on CT-150 and fine-tuned on the 61-scan TCIA training split, updating only background and pancreas output channels.
Scratch (SCR): Models trained from scratch exclusively on the 61-scan TCIA training split.

Regime	Method	Dice Score	Precision	Recall	S2S Dist (mm)
BFT	U-Net	$0.690 \pm 0.132$	$0.680 \pm 0.109$	$0.733 \pm 0.190$	$6.389 \pm 3.900$
BFT	Attention U-Net	$\mathbf{0.712 \pm 0.110}$	$0.693 \pm 0.115$	$\mathbf{0.751 \pm 0.149}$	$\mathbf{5.251 \pm 2.551}$
AFT	U-Net	$0.820 \pm 0.043$	$0.824 \pm 0.070$	$0.828 \pm 0.064$	$2.464 \pm 0.529$
AFT	Attention U-Net	$\mathbf{0.831 \pm 0.038}$	$0.825 \pm 0.073$	$\mathbf{0.840 \pm 0.053}$	$\mathbf{2.305 \pm 0.568}$
SCR	U-Net	$0.815 \pm 0.068$	$0.815 \pm 0.105$	$0.826 \pm 0.062$	$2.576 \pm 1.180$
SCR	Attention U-Net	$\mathbf{0.821 \pm 0.057}$	$0.815 \pm 0.093$	$\mathbf{0.835 \pm 0.057}$	$\mathbf{2.333 \pm 0.856}$

In all three training configurations, Attention U-Net outperforms standard U-Net, achieving statistically significant improvements in Dice score and surface-to-surface distance ( $p = .03$ for SCR S2S distance), driven by higher recall rates. Furthermore, 5-fold cross-validation of Attention U-Net on the full TCIA CT-82 dataset yields a pancreas Dice score of $81.48 \pm 6.23\%$ .

Knowl 8 — Pancreas Segmentation Benchmark Comparison with Multi-Stage Cascaded CNNs

data/table

State-of-the-art pancreas segmentation methods commonly employ multi-stage cascaded CNNs (e.g., stage 1 for coarse organ localization / ROI proposal, followed by stage 2 for fine segmentation on the cropped ROI) or 2D FCN + RNN combinations to overcome low tissue contrast and high anatomical shape variability.

Method	Dataset	Pancreas DSC (%)	Train/Test	# Folds
Hierarchical 3D FCN	CT-150	$82.2 \pm 10.2$	Ext/150	-
Dense-Dilated FCN	CT-82 Synapse	$66.0 \pm 10.0$	63/9	5-CV
2D U-Net	CT-82	$75.7 \pm 9.0$	66/16	5-CV
Holistically Nested 2D FCN Stage-1	CT-82	$76.8 \pm 11.1$	62/20	4-CV
Holistically Nested 2D FCN Stage-2	CT-82	$81.2 \pm 7.3$	62/20	4-CV
2D FCN	CT-82	$80.3 \pm 9.0$	62/20	4-CV
2D FCN + Recurrent Network	CT-82	$82.3 \pm 6.7$	62/20	4-CV
Single Model 2D FCN	CT-82	$75.7 \pm 10.5$	62/20	4-CV
Multi-Model 2D FCN	CT-82	$82.2 \pm 5.7$	62/20	4-CV
Attention U-Net (3D, 5-CV)	CT-82	$81.48 \pm 6.23$	-	5-CV
Attention U-Net (3D, Fine-Tuned)	CT-82	$83.1 \pm 3.8$	61/21	-

Standard single-model networks (e.g., 2D U-Net at 75.7% DSC, Single Model 2D FCN at 75.7% DSC) lag multi-model cascaded pipelines (81.2%–82.3% DSC) by 2.0%–4.4% DSC due to false positives in background tissues. The single-model 3D Attention U-Net achieves $81.48 \pm 6.23\%$ under 5-fold cross validation and $83.1 \pm 3.8\%$ when fine-tuned from CT-150, matching or exceeding multi-stage cascaded systems without requiring separate localization models, ROI cropping, or iterative multi-stage training pipelines.

Knowl 9 — Epoch-Wise Attention Map Convergence and Semantic Specialization
empirical result
Visual and qualitative analysis of attention gate coefficients $\alpha^l$ over the course of training reveals distinct optimization and feature selection behaviors:
1. Initial Uniform Distribution: At early epochs (e.g., epochs 3–6), attention coefficients are uniformly distributed across the spatial grid, allowing feature activations from all spatial locations to pass through skip connections.
2. Coarse-to-Fine Localization: As training progresses (e.g., epochs 10 to 150), attention weights gradually converge to suppress background tissues and isolate targeted anatomical regions. Coarser-scale AGs provide a rough bounding envelope around target organs, which finer-scale AGs progressively refine into sharp organ boundary definitions.
3. Sub-Gate Semantic Specialization: When multiple attention heads (multi-dimensional AGs) are instantiated at a given scale, individual gates autonomously specialize in focusing on different anatomical structures (e.g., distinct gates within the same layer isolating pancreas, kidney, or spleen) consistently across unseen test patient volumes.

Coverage note — None was omitted; all primary contributions, mathematical formulations, architectural details, experimental results, and capacity comparisons are covered.

References

1.Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and vqa. arXiv preprint arXiv:1707.07998 (2017)
2.Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
3.Bai, W., Sinclair, M., Tarroni, G., Oktay, O., Rajchl, M., Vaillant, G., Lee, A.M., Aung, N., Lukaschuk, E., Sanghvi, M.M., et al.: Human-level CMR image analysis with deep fully convolutional networks. arXiv preprint arXiv:1710.09289 (2017)
4.Cai, J., Lu, L., Xie, Y., Xing, F., Yang, L.: Improving deep pancreas segmentation in CT and MRI images via recurrent neural contextual learning and direct loss function. In: MICCAI (2017)
5.Cerrolaza, J.J., Summers, R.M., Linguraru, M.G.: Soft multi-organ shape models via generalized PCA: A general framework. In: MICCAI. pp. 219–228. Springer (2016)
6.Gibson, E., Giganti, F., Hu, Y., Bonmati, E., Bandula, S., Gurusamy, K., Davidson, B.R., Pereira, S.P., Clarkson, M.J., Barratt, D.C.: Towards image-guided pancreas and biliary endoscopy: Automatic multi-organ segmentation on abdominal CT with dense dilated networks. In: MICCAI. pp. 728–736. Springer (2017)
7.Greff, K., Srivastava, R.K., Schmidhuber, J.: Highway and residual networks learn unrolled iterative estimation. arXiv preprint arXiv:1612.07771 (2016)
8.Heinrich, M.P., Blendowski, M., Oktay, O.: TernaryNet: Faster deep model inference without GPUs for medical 3D segmentation using sparse and binary convolutions. arXiv preprint arXiv:1801.09449 (2018)
9.Heinrich, M.P., Oktay, O.: BRIEFnet: Deep pancreas segmentation using binary sparse convolutions. In: MICCAI. pp. 329–337. Springer (2017)
10.Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. arXiv:1709.01507 (2017)
11.Jetley, S., Lord, N.A., Lee, N., Torr, P.: Learn to pay attention. In: International Conference on Learning Representations (2018), https://openreview.net/forum?id=HyzbhfWRW
12.Kamnitsas, K., Bai, W., Ferrante, E., McDonagh, S., Sinclair, M., Pawlowski, N., Rajchl, M., Lee, M., Kainz, B., Rueckert, D., Glocker, B.: Ensembles of multiple models and architectures for robust brain tumour segmentation. In: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. pp. 450–462. Cham (2018)
13.Kamnitsas, K., Ledig, C., Newcombe, V.F., Simpson, J.P., Kane, A.D., Menon, D.K., Rueckert, D., Glocker, B.: Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical image analysis 36, 61–78 (2017)
14.Khened, M., Kollerathu, V.A., Krishnamurthi, G.: Fully convolutional multi-scale residual densenets for cardiac segmentation and automated cardiac diagnosis using ensemble of classifiers. arXiv preprint arXiv:1801.05173 (2018)
15.Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
16.Lee, C.Y., Xie, S., Gallagher, P., Zhang, Z., Tu, Z.: Deeply-supervised nets. In: Artificial Intelligence and Statistics. pp. 562–570 (2015)
17.Liao, F., Liang, M., Li, Z., Hu, X., Song, S.: Evaluate the malignancy of pulmonary nodules using the 3D deep leaky noisy-or network. arXiv preprint arXiv:1711.08324 (2017)
18.Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: IEEE CVPR. pp. 3431–3440 (2015)
19.Luong, M.T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 (2015)
20.Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 3D Vision. pp. 565–571. IEEE (2016)
21.Mnih, V., Heess, N., Graves, A., et al.: Recurrent models of visual attention. In: Advances in neural information processing systems. pp. 2204–2212 (2014)
22.Oda, M., Shimizu, N., Roth, H.R., Karasawa, K., Kitasaka, T., Misawa, K., Fujiwara, M., Rueckert, D., Mori, K.: 3D FCN feature driven regression forest-based pancreas localization and segmentation. In: DLMI, pp. 222–230. Springer (2017)
23.Payer, C., Štern, D., Bischof, H., Urschler, M.: Multi-label whole heart segmentation using CNNs and anatomical label configurations. In: STACOM. pp. 190–198. Springer (2017)
24.Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI. pp. 234–241. Springer (2015)
25.Roth, H., Farag, A., Turkbey, E.B., Lu, L., Liu, J., Summers, R.M.: Data from Pancreas-CT. The Cancer Imaging Archive (2016), http://doi.org/10.7937/K9/TCIA.2016.tNB1kqBU
26.Roth, H.R., Lu, L., Lay, N., Harrison, A.P., Farag, A., Sohn, A., Summers, R.M.: Spatial aggregation of holistically-nested convolutional neural networks for automated pancreas localization and segmentation. Medical Image Analysis 45, 94 – 107 (2018)
27.Roth, H.R., Oda, H., Hayashi, Y., Oda, M., Shimizu, N., Fujiwara, M., Misawa, K., Mori, K.: Hierarchical 3D fully convolutional networks for multi-organ segmentation. arXiv preprint arXiv:1704.06382 (2017)
28.Saito, A., Nawano, S., Shimizu, A.: Joint optimization of segmentation and shape prior from level-set-based statistical shape model, and its application to the automated segmentation of abdominal organs. Medical image analysis 28, 46–65 (2016)
29.Shen, T., Zhou, T., Long, G., Jiang, J., Pan, S., Zhang, C.: Disan: Directional self-attention network for rnn/cnn-free language understanding. arXiv preprint arXiv:1709.04696 (2017)
30.Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: NIPS. pp. 6000–6010 (2017)
31.Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., Bengio, Y.: Graph attention networks. arXiv preprint arXiv:1710.10903 (2017)
32.Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., Tang, X.: Residual attention network for image classification. In: IEEE CVPR. pp. 3156–3164 (2017)
33.Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. arXiv preprint arXiv:1711.07971 (2017)
34.Wolz, R., Chu, C., Misawa, K., Fujiwara, M., Mori, K., Rueckert, D.: Automated abdominal multi-organ segmentation with subject-specific atlas generation. IEEE TMI 32(9) (2013)
35.Xie, S., Tu, Z.: Holistically-nested edge detection. In: Proceedings of the IEEE international conference on computer vision. pp. 1395–1403 (2015)
36.Ypsilantis, P.P., Montana, G.: Learning what to look in chest X-rays with a recurrent visual attention model. arXiv preprint arXiv:1701.06452 (2017)
37.Yu, Q., Xie, L., Wang, Y., Zhou, Y., Fishman, E.K., Yuille, A.L.: Recurrent saliency transformation network: Incorporating multi-stage visual cues for small organ segmentation. arXiv preprint arXiv:1709.04518 (2017)
38.Zhou, Y., Xie, L., Shen, W., Wang, Y., Fishman, E.K., Yuille, A.L.: A fixed-point model for pancreas segmentation in abdominal CT scans. In: MICCAI. pp. 693–701. Springer (2017)
39.Zografos, V., Valentinitsch, A., Rempfler, M., Tombari, F., Menze, B.: Hierarchical multi-organ segmentation without registration in 3D abdominal CT images. In: International MICCAI Workshop on Medical Computer Vision. pp. 37–46. Springer (2015)

Citation

MLA

Oktay, O., et al. “Attention U-Net: Learning Where to Look for the Pancreas”. arXiv, 2018, https://doi.org/10.48550/arxiv.1804.03999.

APA

Oktay, O., Schlemper, J., Folgoc, L. L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N. Y., Kainz, B., Glocker, B., & Rueckert, D. (2018). Attention U-Net: Learning Where to Look for the Pancreas. arXiv. https://doi.org/10.48550/arxiv.1804.03999

Chicago

Oktay, O., J. Schlemper, L. L. Folgoc, et al. 2018. “Attention U-Net: Learning Where to Look for the Pancreas”. Preprint, ArXiv. https://doi.org/10.48550/arxiv.1804.03999.

Harvard

Oktay, O. et al. (2018) “Attention U-Net: Learning Where to Look for the Pancreas”. arXiv. Available at: https://doi.org/10.48550/arxiv.1804.03999.

Vancouver

1. Oktay O, Schlemper J, Folgoc LL, et al (2018) Attention U-Net: Learning Where to Look for the Pancreas. https://doi.org/10.48550/arxiv.1804.03999

BibTeX

@misc{https://doi.org/10.48550/arxiv.1804.03999,
  doi = {10.48550/ARXIV.1804.03999},
  url = {https://arxiv.org/abs/1804.03999},
  author = {Oktay, Ozan and Schlemper, Jo and Folgoc, Loic Le and Lee, Matthew and Heinrich, Mattias and Misawa, Kazunari and Mori, Kensaku and McDonagh, Steven and Hammerla, Nils Y and Kainz, Bernhard and Glocker, Ben and Rueckert, Daniel},
  keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {Attention U-Net: Learning Where to Look for the Pancreas},
  publisher = {arXiv},
  year = {2018},
  copyright = {Creative Commons Attribution 4.0 International}
}

Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Attention U-Net: Learning Where to Look for the Pancreas

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Additive Attention Gate for Dense Feature Filtering

Knowl 2 — Attention U-Net Architecture for Volumetric Medical Segmentation

Knowl 3 — Gradient Backpropagation Formulation for Attention Gated Convolutional Layers

Knowl 4 — Training and Implementation Setup for 3D Attention U-Net

Knowl 5 — Multi-Class Abdominal CT Segmentation Performance on the CT-150 Dataset

Knowl 6 — Segmentation Efficiency Comparison: Attention Gates vs Uniform Capacity Increase

Knowl 7 — Pancreas Segmentation Performance on TCIA Pancreas-CT Under Multiple Training Regimes

Knowl 8 — Pancreas Segmentation Benchmark Comparison with Multi-Stage Cascaded CNNs

Knowl 9 — Epoch-Wise Attention Map Convergence and Semantic Specialization

References

Citation

BibTeX

Source Code

Access the Paper