Back

Brain tumor segmentation with Deep Neural Networks

Mohammad HavaeiAxel DavyDavid Warde-FarleyAntoine BiardAaron CourvilleYoshua BengioChris PalPierre-Marc JodoinHugo Larochelle

article2015MedIA3,161 citations

Proposes a multi-path convolutional neural network for MRI brain tumor segmentation that combines local and global context alongside a two-phase training strategy to handle label imbalance, outperforming prior methods on the BRATS benchmark while running over thirty times faster.

Listen

The research paper describes a fully automatic method for segmenting brain tumors, specifically glioblastomas, from MRI scans. These tumors vary widely in location, shape, size, and appearance, and they often blend into surrounding healthy tissue, making accurate delineation difficult with standard imaging. Manual segmentation by radiologists is slow and variable, while existing automated approaches either rely on hand-crafted image features that lack flexibility or run too slowly for routine clinical use. The work addresses this gap at a time when MRI is central to diagnosis, growth tracking, and treatment planning for the roughly 23,000 new brain-cancer cases diagnosed annually in the United States alone.

The authors set out to build and evaluate convolutional neural network architectures that learn both fine local details and broader spatial context directly from multi-modal MRI data, while remaining fast enough for practical deployment. They tested several designs on the fully annotated 2013 BRATS challenge dataset, which includes 30 training cases and separate test and leaderboard sets, using standard Dice, sensitivity, and specificity metrics for the complete tumor, tumor core, and enhancing tumor regions.

The approach centers on two-pathway CNNs that process small patches at high resolution for local features and larger patches for global context, followed by a final convolutional output layer that speeds inference dramatically. Three cascaded variants feed the first network’s probability maps into a second network to capture label dependencies without the computational cost of conditional random fields. A two-phase training procedure first balances the highly skewed class distribution (healthy tissue dominates) and then recalibrates the output layer on the true distribution. Only minimal preprocessing—intensity normalization and bias correction—is applied.

The strongest model, InputCascadeCNN with two-phase training, reached Dice scores of 0.88 for the complete tumor, 0.79 for the core, and 0.73 for the enhancing region on the held-out test set, placing second overall on the 2013 leaderboard and outperforming the prior winning method. The same model processes a full brain volume in roughly three minutes on a single GPU—more than thirty times faster than the previous best published result—while a lighter two-pathway model finishes in 25 seconds. Joint training of local and global pathways, plus the cascade step, measurably reduced false positives and improved boundary accuracy compared with single-path baselines.

These gains matter because faster, more consistent segmentation can shorten planning time for surgery and radiotherapy and support longitudinal monitoring without added radiologist workload. The speed advantage also opens the door to near-real-time use in image-guided procedures. The method still shows lower accuracy on the enhancing-tumor sub-region, where boundaries are especially diffuse, and it processes data slice-by-slice because the 2013 volumes lack consistent resolution in the third dimension. Results are based on a modest number of cases from a single challenge year, so generalization to newer scanners or acquisition protocols remains to be confirmed.

Further work should include validation on larger, multi-year cohorts and exploration of three-dimensional extensions once isotropic data become available. A modest pilot study comparing automated outputs against multiple expert readers on routine clinical scans would clarify whether the observed accuracy is sufficient for decision support.

Ecole Normale SupérieureÉcole PolytechniquePolytechnique MontréalTwitterUniversité de MontréalUniversité de Sherbrookecomputer visionconvolutional neural networkshigh capacityBrain Tumor SegmentationDeep Neural NetworkConvolutional Neural Network
arXiv: 1505.03540
Cover for Brain tumor segmentation with Deep Neural Networks

Abstract

In this paper, we present a fully automatic brain tumor segmentation method based on Deep Neural Networks (DNNs). The proposed networks are tailored to glioblastomas (both low and high grade) pictured in MR images. By their very nature, these tumors can appear anywhere in the brain and have almost any kind of shape, size, and contrast. These reasons motivate our exploration of a machine learning solution that exploits a flexible, high capacity DNN while being extremely efficient. Here, we give a description of different model choices that we've found to be necessary for obtaining competitive performance. We explore in particular different architectures based on Convolutional Neural Networks (CNN), i.e. DNNs specifically adapted to image data.

We present a novel CNN architecture which differs from those traditionally used in computer vision. Our CNN exploits both local features as well as more global contextual features simultaneously. Also, different from most traditional uses of CNNs, our networks use a final layer that is a convolutional implementation of a fully connected layer which allows a 40 fold speed up. We also describe a 2-phase training procedure that allows us to tackle difficulties related to the imbalance of tumor labels. Finally, we explore a cascade architecture in which the output of a basic CNN is treated as an additional source of information for a subsequent CNN. Results reported on the 2013 BRATS test dataset reveal that our architecture improves over the currently published state-of-the-art while being over 30 times faster.

Topics

computer visionconvolutional neural networkshigh capacitylocal featuresegmentation methodsnetwork architecturearchitecture designsdeep belief networks

Keywords

Brain Tumor SegmentationDeep Neural NetworkConvolutional Neural NetworkTwoPathCNNCascaded ArchitectureTwo‑phase TrainingBRATS 2013MRI ModalityMaxoutDropout

Table of Contents

  • Keywords:
  • 1. Introduction
  • 2. Related work
  • 3. Our Convolutional Neural Network Approach
  • 3.1. The Architectures
  • 3.1.1. Two-pathway architecture
  • 3.1.2. Cascaded architectures
  • 3.2. Training
  • 4. Implementation details
  • 5. Experiments and Results
  • 5.1. The TwoPATHCNN architecture
  • 5.2. Cascaded architectures
  • 6. Conclusion
  • References
  • References

Knowls

  1. Knowl 1 — Two-Pathway Convolutional Neural Network Architecture for Brain Tumor Segmentation

    model/method

    The two-pathway convolutional neural network (TwoPathCNN) is designed to segment brain tumors from multi-sequence 2D axial magnetic resonance imaging (MRI) slices by simultaneously extracting fine local visual details and coarse global anatomical context.

    The input to the network is a 2D axial patch of dimensions 33×3333 \times 33 across R=4R = 4 co-registered MRI channels: T1, T1-contrast enhanced (T1C), T2, and Fluid Attenuation Inversion Recovery (FLAIR).

    The network comprises two parallel pathways:

    1. Local Pathway: Captures fine-grained morphological details using two successive convolutional stages with valid convolutions:
    • First layer: 7×77 \times 7 convolutions with K=2K = 2 Maxout units, followed by 4×44 \times 4 max-pooling with stride S=1S = 1. This maps the 33×3333 \times 33 input to 64 feature maps of spatial size 24×2424 \times 24.
    • Second layer: 3×33 \times 3 convolutions with K=2K = 2 Maxout units, followed by 2×22 \times 2 max-pooling with stride S=1S = 1. This transforms the 24×2424 \times 24 maps into 64 feature maps of spatial size 21×2121 \times 21.
    1. Global Pathway: Captures broader spatial context (the relative location within the brain) using a single convolutional stage without pooling:
    • Convolutional layer: Large 13×1313 \times 13 kernels with K=2K = 2 Maxout units. Applied directly to the 33×3333 \times 33 input, it yields 160 feature maps of spatial size 21×2121 \times 21.

    Feature Concatenation and Output: The 64 local feature maps and 160 global feature maps are concatenated along the channel dimension into a single 224×21×21224 \times 21 \times 21 feature representation. A final convolutional layer with 21×2121 \times 21 kernels and a softmax non-linearity maps this representation to a 5-dimensional probability vector (5×1×15 \times 1 \times 1), predicting the class distribution for the central pixel of the input patch across five categories: non-tumor, necrosis, edema, non-enhancing tumor, and enhancing tumor. The architecture contains 651,488 trainable parameters.

  2. Knowl 2 — Two-Phase Training Algorithm for Imbalanced Multiclass Brain Tumor Segmentation

    algorithm

    Brain MRI tumor segmentation suffers from severe class imbalance: approximately 98%98\% of voxels represent healthy tissue (class 0), while pathological classes account for only 2%2\% (0.18%0.18\% necrosis [class 1], 1.1%1.1\% edema [class 2], 0.12%0.12\% non-enhancing tumor [class 3], and 0.38%0.38\% enhancing tumor [class 4]). Directly training a deep network on randomly sampled patches causes healthy tissue to dominate gradient updates, leading to poor tumor feature representations and failure to detect rare sub-classes.

    The two-phase training procedure addresses this imbalance:

    Input: Training dataset of 2D multi-sequence MRI slices with voxel-level ground truth labels Y∈{0,1,2,3,4}Y \in \{0, 1, 2, 3, 4\}, CNN architecture with feature extraction layers Θconv\Theta_{\text{conv}} and output classification layer Θout\Theta_{\text{out}}
    Output: Trained CNN model parameters (Θconv∗,Θout∗)(\Theta_{\text{conv}}^*, \Theta_{\text{out}}^*)
    // Phase 1: Feature learning with balanced label distribution
    Construct dataset Dphase1\mathcal{D}_{\text{phase1}} by sampling 2D patches such that center labels are equiprobable:
        p(Y=0)=p(Y=1)=p(Y=2)=p(Y=3)=p(Y=4)=0.20p(Y = 0) = p(Y = 1) = p(Y = 2) = p(Y = 3) = p(Y = 4) = 0.20
    Initialize all kernel weights randomly from U(−0.005,0.005)U(-0.005, 0.005) and biases to 0
    Train all network parameters (Θconv,Θout)(\Theta_{\text{conv}}, \Theta_{\text{out}}) jointly on Dphase1\mathcal{D}_{\text{phase1}} via SGD with momentum until validation performance ceases to improve:
        Minimize LNLL(Θconv,Θout)+λ1∥Θ∥1+λ2∥Θ∥22\mathcal{L}_{\text{NLL}}(\Theta_{\text{conv}}, \Theta_{\text{out}}) + \lambda_1 \|\Theta\|_1 + \lambda_2 \|\Theta\|_2^2
    Set Θconv∗←Θconv\Theta_{\text{conv}}^* \leftarrow \Theta_{\text{conv}}
    // Phase 2: Calibration of output layer on representative distribution
    Freeze all convolutional feature extraction parameters Θconv∗\Theta_{\text{conv}}^*
    Construct dataset Dphase2\mathcal{D}_{\text{phase2}} by sampling patches according to the empirical class frequencies (98% healthy, 2% tumorous tissues)
    Re-initialize output layer biases to the logarithm of the natural class frequencies
    Train only the final classification layer parameters Θout\Theta_{\text{out}} on Dphase2\mathcal{D}_{\text{phase2}} via SGD with momentum until convergence:
        Minimize LNLL(Θout;Θconv∗)\mathcal{L}_{\text{NLL}}(\Theta_{\text{out}}; \Theta_{\text{conv}}^*)
    Set Θout∗←Θout\Theta_{\text{out}}^* \leftarrow \Theta_{\text{out}}
    return (Θconv∗,Θout∗)(\Theta_{\text{conv}}^*, \Theta_{\text{out}}^*)

    Phase 1 enables the lower and intermediate feature extractors to allocate model capacity evenly across all classes, while Phase 2 calibrates output probabilities to match empirical prior frequencies, eliminating the high false-positive rate that results from equiprobable training alone.

  3. Knowl 3 — Cascaded CNN Architectures for Spatial Label Dependency Modeling

    model/method

    Standard convolutional networks predict each voxel's label conditionally independent of neighboring labels given the input image. Rather than using computationally expensive Conditional Random Field (CRF) inference to model spatial label dependencies, cascaded architectures feed the pixel-wise probability predictions of a first trained CNN as additional contextual input channels into a second CNN.

    Three cascaded configurations are defined:

    1. Input Concatenation (InputCascadeCNN): The first CNN takes a larger patch of size 65×65×465 \times 65 \times 4 and outputs a 5×33×335 \times 33 \times 33 label probability map. This 5-channel map is concatenated directly with the original 4×33×334 \times 33 \times 33 raw MRI patch, providing a 9-channel input of size 9×33×339 \times 33 \times 33 to the second two-pathway CNN. Total parameters: 802,368; average whole-brain inference time: 3.0 minutes.

    2. Local Pathway Concatenation (LocalCascadeCNN): The first CNN takes an input patch of size 56×56×456 \times 56 \times 4 and outputs a 5×24×245 \times 24 \times 24 probability map. This is concatenated with the 64 feature maps produced by the first hidden layer of the second CNN's local pathway, forming a 69-channel input (69×24×2469 \times 24 \times 24) to the second local convolution layer. Total parameters: 654,368; average whole-brain inference time: 1.7 minutes.

    3. Pre-Output Concatenation (MFCascadeCNN): The first CNN processes a 53×53×453 \times 53 \times 4 input patch to generate a 5×21×215 \times 21 \times 21 probability map. This is concatenated directly with the top 224×21×21224 \times 21 \times 21 merged local-global features of the second CNN, yielding a 229-channel input (229×21×21229 \times 21 \times 21) to the final 21×2121 \times 21 output convolution layer. This structure mirrors one pass of CRF mean-field message passing where pairwise potentials correspond to convolutional output weights. Total parameters: 662,513; average whole-brain inference time: 1.5 minutes.

    In all variants, the first CNN is trained using two-phase training and frozen; the second CNN is then trained on top of its outputs using the same two-phase training protocol.

  4. Knowl 4 — Fully Convolutional Implementation for Accelerated Whole-Brain Inference

    model/method

    Traditional patch-based CNNs evaluate individual M×MM \times M patches centered at each pixel via dense matrix multiplications in a fully connected output layer. Processing every voxel in a 3D brain scan sequentially creates massive computational redundancy and slow runtime.

    To achieve rapid inference:

    1. The final classification layer is implemented as a valid-mode 2D convolution with kernel size 21×2121 \times 21 followed by a softmax non-linearity across the 5 class channels, rather than a flattened fully connected layer.
    2. All max-pooling operations use a stride of S=1S = 1 to preserve dense pixel-grid alignment without downsampling the resolution.
    3. At test time, an entire 2D axial slice of dimensions H×W×4H \times W \times 4 is passed directly as a single input tensor to the network. Convolutional operations at all layers process the entire slice simultaneously in a feed-forward pass on the GPU, evaluating all pixel label likelihoods p(Yij∣X)p(Y_{ij} \mid X) concurrently.

    This fully convolutional approach achieves a 45-fold speedup over pixel-by-pixel patch extraction, segmenting an entire 3D brain volume (approximately 200 axial slices) in 25 seconds for the TwoPathCNN and 1.5 to 3 minutes for cascaded architectures on an NVIDIA Titan Black GPU.

  5. Knowl 5 — Mathematical Formulation of Maxout Convolutional Layers and Regularized Multiclass Objective

    equation

    In a convolutional layer with RR input planes XrX_r, the ss-th linear pre-activation feature map OsO_s is defined by valid 2D convolution:

    Os=bs+∑r=1RWsr∗XrO_s = b_s + \sum_{r=1}^R W_{sr} * X_r

    where Wsr∈RN×NW_{sr} \in \mathbb{R}^{N \times N} is the sub-kernel for input channel rr, ∗* is the discrete 2D valid convolution operator, and bs∈Rb_s \in \mathbb{R} is a scalar bias.

    Maxout Activation: For each group of KK consecutive linear feature maps {Os,Os+1,…,Os+K−1}\{O_s, O_{s+1}, \dots, O_{s+K-1}\}, the Maxout unit computes the point-wise maximum across the KK maps at each spatial location (i,j)(i, j):

    Zs,i,j=max⁡k∈{0,1,…,K−1}Os+k,i,jZ_{s, i, j} = \max_{k \in \{0, 1, \dots, K-1\}} O_{s+k, i, j}

    Max-Pooling: Over a spatial window of size p×pp \times p with stride SS, max-pooling produces feature map HsH_s with spatial dimensions D×DD \times D, where D=(Q−p)/S+1D = (Q - p)/S + 1 from an input of size Q×QQ \times Q:

    Hs,i,j=max⁡p1,p2∈{0,1,…,p−1}Zs,i⋅S+p1,j⋅S+p2H_{s, i, j} = \max_{p_1, p_2 \in \{0, 1, \dots, p-1\}} Z_{s, i \cdot S + p_1, j \cdot S + p_2}

    Multiclass Softmax Likelihood: Given the output activation vector a∈RCa \in \mathbb{R}^C (C=5C = 5) at spatial position (i,j)(i, j), the predicted probability for class cc is:

    p(Yij=c∣X)=exp⁡(ac)∑k=1Cexp⁡(ak)p(Y_{ij} = c \mid X) = \frac{\exp(a_c)}{\sum_{k=1}^C \exp(a_k)}

    Regularized Training Objective: The total loss function combines negative log-likelihood with L1L_1 and L2L_2 weight decay regularization:

    L(W)=−∑i,jlog⁡p(Yij∣X)+λ1∥W∥1+λ2∥W∥22\mathcal{L}(W) = -\sum_{i, j} \log p(Y_{ij} \mid X) + \lambda_1 \|W\|_1 + \lambda_2 \|W\|_2^2

    where ∥W∥1=∑w∣w∣\|W\|_1 = \sum_w |w| encourages sparsity, and ∥W∥22=∑ww2\|W\|_2^2 = \sum_w w^2 penalizes large weights. Optimization uses momentum SGD with velocity updates:

    Vt+1=μVt−α∇WtL(Wt),Wt+1=Wt+Vt+1V_{t+1} = \mu V_t - \alpha \nabla_{W_t} \mathcal{L}(W_t), \quad W_{t+1} = W_t + V_{t+1}

    where learning rate α\alpha is initialized at 0.0050.005 and decayed by 10−110^{-1} per epoch, and momentum coefficient μ\mu is scheduled from 0.50.5 to 0.90.9. Dropout is applied with masking probability 0.50.5 on hidden layers.

  6. Knowl 6 — Tumor Region Definitions and Segmentation Evaluation Metrics

    definition

    Clinical evaluation of brain tumor segmentation decomposes five target tissue classes—non-tumor (0), necrosis (1), edema (2), non-enhancing tumor (3), and enhancing tumor (4)—into three nested sub-regions:

    1. Complete Tumor Region: Encompasses all pathological tumor structures: Complete={1,2,3,4}\text{Complete} = \{1, 2, 3, 4\}
    2. Core Tumor Region: Encompasses the active and necrotic tumor mass, excluding edema: Core={1,3,4}\text{Core} = \{1, 3, 4\}
    3. Enhancing Tumor Region: Encompasses only the vascularized, contrast-enhancing tumor tissue: Enhancing={4}\text{Enhancing} = \{4\}

    Let P1P_1 and T1T_1 denote the set of voxels predicted as positive and ground truth positive for a specified region, respectively. Let P0P_0 and T0T_0 denote the corresponding sets of negative voxels.

    Evaluation is measured using three voxel-level metrics:

    • Dice Score (F-Measure): Dice(P,T)=2∣P1∩T1∣∣P1∣+∣T1∣\text{Dice}(P, T) = \frac{2 |P_1 \cap T_1|}{|P_1| + |T_1|}

    • Sensitivity (True Positive Rate): Sensitivity(P,T)=∣P1∩T1∣∣T1∣\text{Sensitivity}(P, T) = \frac{|P_1 \cap T_1|}{|T_1|}

    • Specificity (True Negative Rate): Specificity(P,T)=∣P0∩T0∣∣T0∣\text{Specificity}(P, T) = \frac{|P_0 \cap T_0|}{|T_0|}

    Multimodal preprocessing comprises three steps: (1) trimming the 1%1\% highest and lowest intensity outliers, (2) applying N4ITK bias field correction to T1 and T1C volumes, and (3) channel-wise zero-mean unit-variance intensity normalization. Post-processing uses connected components to discard flat isolated false positives located near bright skull edges.

  7. Knowl 7 — Comparative Performance of Brain Tumor Segmentation Models on BRATS 2013 Test Set

    data/table

    Evaluation on the BRATS 2013 test dataset (10 patient subjects with high-grade gliomas) compares the proposed single-path, two-pathway, and cascaded CNN architectures (trained with two-phase training, denoted by *) against prior published state-of-the-art methods.

    Method Dice Specificity Sensitivity
    Complete Core Enhancing Complete Core Enhancing Complete Core Enhancing
    InputCascadeCNN* 0.88 0.79 0.73 0.89 0.79 0.68 0.87 0.79 0.80
    Tustison (BRATS 2013 Winner) 0.87 0.78 0.74 0.85 0.74 0.69 0.89 0.88 0.83
    MFCascadeCNN* 0.86 0.77 0.73 0.92 0.80 0.71 0.81 0.76 0.76
    TwoPathCNN* 0.85 0.78 0.73 0.93 0.80 0.72 0.80 0.76 0.75
    LocalCascadeCNN* 0.88 0.76 0.72 0.91 0.76 0.70 0.84 0.80 0.75
    LocalPathCNN* 0.85 0.74 0.71 0.91 0.75 0.71 0.80 0.77 0.73
    Meier 0.82 0.73 0.69 0.76 0.78 0.71 0.92 0.72 0.73
    Reza 0.83 0.72 0.72 0.82 0.81 0.70 0.86 0.69 0.76
    Zhao 0.84 0.70 0.65 0.80 0.67 0.65 0.89 0.79 0.70
    Cordier 0.84 0.68 0.65 0.88 0.63 0.68 0.81 0.82 0.66
    TwoPathCNN (1-phase) 0.78 0.63 0.68 0.67 0.50 0.59 0.96 0.89 0.82
    LocalPathCNN (1-phase) 0.77 0.64 0.68 0.65 0.52 0.60 0.96 0.87 0.80
    Festa 0.72 0.66 0.67 0.77 0.77 0.70 0.72 0.60 0.70
    Doyle 0.71 0.46 0.52 0.66 0.38 0.58 0.87 0.70 0.55

    InputCascadeCNN* achieves top performance on Complete (0.880.88) and Core (0.790.79) tumor Dice scores, surpassing Tustison et al. (the challenge winner). Crucially, InputCascadeCNN* requires 3 minutes per brain on an NVIDIA Titan Black GPU compared to 100 minutes for Tustison et al., yielding a >30-fold speedup. The uncascaded TwoPathCNN* model segments a brain in 25 seconds (>200-fold speedup) while remaining highly competitive (Dice: 0.85 Complete, 0.78 Core, 0.73 Enhancing).

  8. Knowl 8 — Ablation Analysis of Pathway Design and Two-Phase Training

    data/table

    The ablation study isolates the contributions of pathway coupling (local vs. global vs. two-pathway) and training protocols (single-phase vs. two-phase) on the BRATS 2013 online test evaluation benchmark. Models trained with two-phase training are designated with an asterisk (*).

    Rank Method Dice Specificity Sensitivity
    Complete Core Enhancing Complete Core Enhancing Complete Core Enhancing
    4 TwoPathCNN* 0.85 0.78 0.73 0.93 0.80 0.72 0.80 0.76 0.75
    9 LocalPathCNN* 0.85 0.74 0.71 0.91 0.75 0.71 0.80 0.77 0.73
    10 AverageCNN* 0.84 0.75 0.70 0.95 0.83 0.73 0.77 0.74 0.73
    14 GlobalPathCNN* 0.82 0.73 0.68 0.93 0.81 0.70 0.75 0.65 0.70
    14 TwoPathCNN 0.78 0.63 0.68 0.67 0.50 0.59 0.96 0.89 0.82
    15 LocalPathCNN 0.77 0.64 0.68 0.65 0.52 0.60 0.96 0.87 0.80

    Key empirical conclusions:

    1. Effect of Two-Phase Training: Retraining the output layer on the natural class distribution drastically improves performance across all architectures. For LocalPathCNN, rank increases from 15 to 9, and Specificity jumps from 0.650.65 to 0.910.91 (Complete) and 0.520.52 to 0.750.75 (Core). For TwoPathCNN, rank increases from 14 to 4, and Complete Dice rises from 0.780.78 to 0.850.85.
    2. Joint Training vs. Ensembling: Jointly training the two pathways (TwoPathCNN*, rank 4) outperforms averaging the independent outputs of separately trained local and global models (AverageCNN*, rank 10, Dice: 0.84/0.75/0.70), demonstrating that end-to-end backpropagation enables the local and global pathways to co-adapt complementary feature hierarchies.

  9. Knowl 9 — Comparative Performance of Cascaded CNN Architecture Configurations

    data/table

    The three cascaded architecture variants—concatenating the first CNN's probability outputs at the input layer (InputCascadeCNN*), at the first hidden local layer (LocalCascadeCNN*), or immediately before the output layer (MFCascadeCNN*)—were evaluated on the BRATS 2013 test dataset using two-phase training.

    Rank Method Dice Specificity Sensitivity
    Complete Core Enhancing Complete Core Enhancing Complete Core Enhancing
    2 InputCascadeCNN* 0.88 0.79 0.73 0.89 0.79 0.68 0.87 0.79 0.80
    4-a MFCascadeCNN* 0.86 0.77 0.73 0.92 0.80 0.71 0.81 0.76 0.76
    4-c LocalCascadeCNN* 0.88 0.76 0.72 0.91 0.76 0.70 0.84 0.80 0.75

    Findings:

    • InputCascadeCNN* achieves the best overall performance, ranking 2nd on the online scoreboard and improving Dice metrics across all three tumor subregions (Complete: 0.880.88, Core: 0.790.79, Enhancing: 0.730.73).
    • MFCascadeCNN* achieves higher specificity and yields visually smoother boundaries between tumor sub-classes because its direct connection from previous probability outputs to final output neurons biases adjacent predictions toward spatial consensus (analogous to CRF mean-field iterations).
    • LocalCascadeCNN* reduces false positives in the Complete tumor category but shows no improvement in Core or Enhancing regions.

  10. Knowl 10 — Generalization Performance on BRATS 2013 Leaderboard and BRATS 2012 Test Datasets

    data/table

    The generalization capability of InputCascadeCNN* was validated on the BRATS 2013 Leaderboard dataset (25 patient subjects: 21 high grade and 4 low grade gliomas) and the BRATS 2012 "4 label" test dataset.

    BRATS 2013 Leaderboard Dataset:

    Method Dice Specificity Sensitivity
    Complete Core Enhancing Complete Core Enhancing Complete Core Enhancing
    InputCascadeCNN* 0.84 0.71 0.57 0.88 0.79 0.54 0.84 0.72 0.68
    Tustison 0.79 0.65 0.53 0.83 0.70 0.51 0.81 0.73 0.66
    Zhao 0.79 0.59 0.47 0.77 0.55 0.50 0.85 0.77 0.53
    Meier 0.72 0.60 0.53 0.65 0.62 0.48 0.88 0.69 0.60
    Reza 0.73 0.56 0.51 0.68 0.64 0.48 0.79 0.57 0.63
    Cordier 0.75 0.61 0.46 0.79 0.61 0.43 0.78 0.72 0.52

    BRATS 2012 "4 Label" Test Dataset:

    Method Dice
    Complete Core Enhancing
    InputCascadeCNN* 0.81 0.72 0.58
    Subbanna 0.75 0.70 0.59
    Zhao 0.82 0.66 0.42
    Tustison 0.75 0.55 0.52
    Festa 0.62 0.50 0.61

    On the BRATS 2013 Leaderboard set, InputCascadeCNN* outperforms all competing methods across all three tumor categories in Dice, Specificity, and Sensitivity. On the BRATS 2012 benchmark, it achieves the highest Core tumor Dice (0.720.72) and competitive Complete (0.810.81) and Enhancing (0.580.58) scores.

Coverage note — BRATS 2015 challenge submission results (shown only as boxplot distributions in Figure 10) are omitted from individual tabular knowls because exact numerical performance tables were not publicly available or reported in tabular form in the paper.

References

  1. 1.Alvarez, J.M., Gevers, T., LeCun, Y., Lopez, A.M., 2012. Road scene segmentation from a single image, in: Proceedings of the 12th European Conference on Computer Vision - Volume Part VII, Springer-Verlag, Berlin, Heidelberg. pp. 376–389.
  2. 2.Angelini, E., Clatz, O., E., Konukoglu, E., Capelle, L., Duffau, H., 2007. Glioma dynamics and computational models: A review of segmentation, registration, and in silico growth algorithms and their clinical applications 3.
  3. 3.Avants, B.B., Tustison, N., Song, G., 2009. Advanced normalization tools (ants). Insight J .
  4. 4.Bauer, S., Nolte, L.P., Reyes, M., 2011. Fully automatic segmentation of brain tumor images using support vector machine classification in combination with hierarchical conditional random field regularization., in: MICCAI, pp. 354–361.
  5. 5.Bauer, S., Wiest, R., Nolte, L., Reyes, M., 2013. A survey of mri-based medical image analysis for brain tumor studies. Physics in medicine and biology 58, 97–129.
  6. 6.Bengio, Y., 2012. Practical recommendations for gradient-based training of deep architectures, in: Neural Networks: Tricks of the Trade. Springer, pp. 437–478.
  7. 7.Bengio, Y., Courville, A., Vincent, P., 2013. Representation learning: A review and new perspectives. Pattern Analysis and Machine Intelligence, IEEE Transactions on 35, 1798–1828.
  8. 8.Ciresan, D., Giusti, A., Gambardella, L.M., Schmidhuber, J., 2012. Deep neural networks segment neuronal membranes in electron microscopy images, in: Advances in neural information processing systems, pp. 2843–2851.
  9. 9.Clark, M., Hall, L., Goldgof, D., Velthuizen, R.P., Murtagh, F., Silbiger, M.L., 1998. Automatic tumor segmentation using knowledge-based clustering. IEEE Trans. Med. Imaging 17, 187–201.
  10. 10.Cobzas, D., Birkbeck, N., Schmidt, M., Jgersand, M., Murtha, A., 2007. 3d variational brain tumor segmentation using a high dimensional feature set, in: ICCV, pp. 1–8.
  11. 11.Davy, A., Havaei, M., Warde-Farley, D., Biard, A., Tran, L., Jodoin, P.M., Courville, A., Larochelle, H., Pal, C., Bengio, Y., 2014. Brain tumor segmentation with deep neural networks. in proc of BRATS-MICCAI .
  12. 12.Doyle, S., Vasseur, F., Dojat, M., Forbes, F., 2013. Fully automatic brain tumor segmentation from multiple mr sequences using hidden markov fields and variational em. in proc of BRATS-MICCAI .
  13. 13.Farabet, C., Couprie, C., Najman, L., LeCun, Y., 2013. Learning hierarchical features for scene labeling. Pattern Analysis and Machine Intelligence, IEEE Transactions on 35, 1915–1929.
  14. 14.Farahani, K., Menze, B., Reyes, M., 2013. Multimodal Brain Tumor Segmentation (BRATS 2013). URL: http://martinos.org/qtim/miccai2013/.
  15. 15.Farahani, K., Menze, B., Reyes, M., 2014. Brats 2014 Challenge Manuscripts. URL: http://www.braintumorsegmentation.org.
  16. 16.Glorot, X., Bordes, A., Bengio, Y., 2011. Domain adaptation for large-scale sentiment classification: A deep learning approach, in: Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 513–520.
  17. 17.Goodfellow, I.J., Warde-Farley, D., Lamblin, P., Dumoulin, V., Mirza, M., Pascanu, R., Bergstra, J., Bastien, F., Bengio, Y., 2013a. Pylearn2: a machine learning research library. arXiv preprint arXiv:1308.4214 .
  18. 18.Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y., 2013b. Maxout networks, in: ICML.
  19. 19.Gotz, M., Weber, C., Blocher, J., Stieltjes, B., Meinzer, H.P., Maier-Hein, K., 2014. Extremely randomized trees based brain tumor segmentation, in: in proc of BRATS Challenge - MICCAI.
  20. 20.Hamamci, A., Kucuk, N., Karaman, K., Engin, K., Unal, G., 2012. Tumor-cut: Segmentation of brain tumors on contrast enhanced mr images for radiosurgery applications. IEEE trans. Medical Imaging 31, 790–804.
  21. 21.Hariharan, B., Arbelaez, P., Girshick, R., Malik, J., 2014. Simultaneous detection and segmentation, in: Computer Vision–ECCV 2014. Springer, pp. 297–312.
  22. 22.Havaei, M., Jodoin, P.M., Larochelle, H., 2014. Efficient interactive brain tumor segmentation as within-brain knn classification, in: International Conference on Pattern Recognition (ICPR).
  23. 23.Huang, G.B., Jain, V., 2013. Deep and wide multiscale recursive networks for robust image labeling. arXiv preprint arXiv:1310.0354 .
  24. 24.Jarrett, K., Kavukcuoglu, K., Ranzato, M., LeCun, Y., 2009. What is the best multi-stage architecture for object recognition?, in: Computer Vision, 2009 IEEE 12th International Conference on, IEEE. pp. 2146–2153.
  25. 25.Khotanlou, H., Colliot, O., Atif, J., Bloch, I., 2009. 3d brain tumor segmentation in mri using fuzzy classification, symmetry analysis and spatially constrained deformable models. Fuzzy Sets Syst. 160, 1457–1473.
  26. 26.Kleesiek, J., Biller, A., Urban, G., Kothe, U., Bendszus, M., Hamprecht, F.A., 2014. ilastik for multi-modal brain tumor segmentation. in proc of BRATS-MICCAI .
  27. 27.Krizhevsky, A., Sutskever, I., Hinton, G., 2012. ImageNet classification with deep convolutional neural networks, in: NIPS.
  28. 28.Kwon, D., Akbari, H., Da, X., Gaonkar, B., Davatzikos, C., 2014. Multi-modal brain tumor image segmentation using glistr, in: in proc of BRATS Challenge - MICCAI.
  29. 29.LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 2278–2324.
  30. 30.Lee, C.H., Schmidt, M., Murtha, A., Bistritz, A., S, J., Greiner, R., 2005. Segmenting brain tumor with conditional random fields and support vector machines, in: in Proc of Workshop on Computer Vision for Biomedical Image Applications.
  31. 31.Long, J., Shelhamer, E., Darrell, T., 2015. Fully convolutional networks for semantic segmentation. CVPR (to appear) .
  32. 32.Menze, B., Reyes, M., Leemput, K.V., 2014. The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans. on Medical Imaging (accepted) .
  33. 33.N.Tustison, M. Wintermark, C.D., Avants, B., 2013. Ants and arboles, in: in proc of BRATS Challenge - MICCAI.
  34. 34.Parisot, S., Duffau, H., Chemouny, S., Paragios, N., 2012. Joint tumor segmentation and dense deformable registration of brain mr images., in: MICCAI, pp. 651–658.
  35. 35.Pinheiro, P., Collobert, R., 2014. Recurrent convolutional neural networks for scene labeling, in: Proceedings of The 31st International Conference on Machine Learning, pp. 82–90.
  36. 36.Popuri, K., Cobzas, D., Murtha, A., Jgersand, M., 2012. 3d variational brain tumor segmentation using dirichlet priors on a clustered feature set. Int. J. Computer Assisted Radiology and Surgery 7, 493–506.
  37. 37.Prastawa, M., Bullit, E., Ho, S., Gerig, G., 2004. A brain tumor segmentation framework based on outlier detection. Medical Image Anaylsis 8, 275–283.
  38. 38.Prastawa, M., Bullitt, E., Ho, S., Gerig, G., 2003. Robust estimation for brain tumor segmentation, in: Medical Image Computing and Computer-Assisted Intervention-MICCAI 2003. Springer, pp. 530–537.
  39. 39.R.Meier, S.Bauer, J.Slotboom, R.Wiest, M.Reyes, 2014. Appearance-and context-sensitive features for brain tumor segmentation, in: in proc of BRATS Challenge - MICCAI.
  40. 40.Rumelhart, D.E., Hinton, G.E., Williams, R.J., 1988. Learning representations by back-propagating errors. Cognitive modeling 5.
  41. 41.Schmidt, M., Levner, I., Greiner, R., Murtha, A., Bistritz, A., 2005. Segmenting brain tumors using alignment-based features, in: Int. Conf on Machine Learning and Applications, pp. 6–pp.
  42. 42.Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R., 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15, 1929–1958. URL: http://jmlr.org/papers/v15/srivastava14a.html.
  43. 43.Subbanna, N., Precup, D., Arbel, T., 2014. Iterative multilevel mrf leveraging context and voxel information for brain tumour segmentation in mri.
  44. 44.Subbanna, N., Precup, D., Collins, L., Arbel, T., 2013. Hierarchical probabilistic gabor and mrf segmentation of brain tumours in mri volumes., in: in proc of MICCAI, pp. 751–758.
  45. 45.Urban, G., Bendszus, M., Hamprecht, F., Kleesiek, J., 2014. Multi-modal brain tumor segmentation using deep convolutional neural networks. in proc of BRATS-MICCAI .
  46. 46.Xing, E.P., Jordan, M.I., Russell, S., 2002. A generalized mean field algorithm for variational inference in exponential families, in: Proceedings of the Nineteenth conference on Uncertainty in Artificial Intelligence, Morgan Kaufmann Publishers Inc.. pp. 583–591.
  47. 47.Zeiler, M.D., Fergus, R., 2014. Visualizing and understanding convolutional networks, in: Computer Vision–ECCV 2014. Springer, pp. 818–833.
  48. 48.Zikic, D., Glocker, B., Konukoglu, E., Criminisi, A., Demiralp, C., Shotton, J., Thomas, O., Das, T., Jena, R., Price, S., 2012. Decision forests for tissue-specific segmentation of high-grade gliomas in multi-channel mr, in: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2012. Springer, pp. 369–376.
  49. 49.Zikic, D., Ioannou, Y., Brown, M., Criminisi, A., 2014. Segmentation of brain tumor tissues with convolutional neural networks. in proc of BRATS-MICCAI .

Citation

MLA
Havaei, M., et al. “Brain Tumor Segmentation with Deep Neural Networks”. Medical Image Analysis, vol. 35, 2017, pp. 18–31, https://doi.org/10.1016/j.media.2016.05.004.
APA
Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A., Bengio, Y., Pal, C., Jodoin, P.-M., & Larochelle, H. (2017). Brain tumor segmentation with Deep Neural Networks. Medical Image Analysis, 35, 18–31. https://doi.org/10.1016/j.media.2016.05.004
Chicago
Havaei, M., A. Davy, D. Warde-Farley, et al. 2017. “Brain Tumor Segmentation with Deep Neural Networks”. Medical Image Analysis 35: 18–31. https://doi.org/10.1016/j.media.2016.05.004.
Harvard
Havaei, M. et al. (2017) “Brain tumor segmentation with Deep Neural Networks”, Medical Image Analysis, 35, pp. 18–31. Available at: https://doi.org/10.1016/j.media.2016.05.004.
Vancouver
1. Havaei M, Davy A, Warde-Farley D, Biard A, Courville A, Bengio Y, Pal C, Jodoin P-M, Larochelle H (2017) Brain tumor segmentation with Deep Neural Networks. Medical Image Analysis 35:18–31

BibTeX

@article{Havaei_2017, title={Brain tumor segmentation with Deep Neural Networks}, volume={35}, ISSN={1361-8415}, url={http://dx.doi.org/10.1016/j.media.2016.05.004}, DOI={10.1016/j.media.2016.05.004}, journal={Medical Image Analysis}, publisher={Elsevier BV}, author={Havaei, Mohammad and Davy, Axel and Warde-Farley, David and Biard, Antoine and Courville, Aaron and Bengio, Yoshua and Pal, Chris and Jodoin, Pierre-Marc and Larochelle, Hugo}, year={2017}, month=Jan, pages={18–31} }
Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF