Back

Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics

Alex KendallYarin GalRoberto Cipolla

article2017CVPR4,416 citations

Proposes a principled multi-task learning method that automatically balances classification and regression losses using task-dependent homoscedastic uncertainty, eliminating manual weight tuning and outperforming individually trained models on joint scene understanding.

Listen

The paper addresses the challenge of jointly learning multiple vision tasks—semantic segmentation, instance segmentation, and per-pixel depth regression—from a single monocular image. Performance in such multi-task settings depends heavily on how the individual task losses are weighted, yet manual or uniform weighting is expensive to tune and often yields suboptimal results, especially for tasks with different units and noise characteristics. This limits practical use in real-time systems such as robotics, where separate models for each task would be computationally prohibitive.

The work sets out to derive and validate a principled loss function that automatically learns relative task weights by treating homoscedastic uncertainty as a learnable parameter for each output. The authors start from a maximum-likelihood formulation: for regression outputs they model a Gaussian likelihood whose variance acts as a task weight, and for classification outputs they scale the softmax logits by an analogous uncertainty term. The resulting objective is differentiable, requires no additional hyper-parameter search, and is applied to a shared ResNet-101 encoder with task-specific decoders on the Cityscapes dataset.

Experiments on both down-sampled validation data and the full-resolution benchmark show that the learned weighting consistently outperforms single-task baselines and naive multi-task sums. On the reduced-resolution set the combined model raises semantic-segmentation IoU from 59.4 % to 63.4 % while also lowering depth and instance errors; on the full test set it reaches 78.5 % class IoU, 21.6 % instance AP, and 2.92 px mean depth error with one forward pass. The final learned loss ratio is approximately 43 : 1 : 0.16 for semantic segmentation, depth regression, and instance segmentation, confirming that the network discovers non-uniform, time-varying task importances.

These gains matter because they simultaneously improve accuracy and cut inference cost, enabling richer scene understanding on embedded platforms. The approach also demonstrates that complementary cues across geometry and semantics regularize one another, an effect that separate models cannot exploit. Because the weighting emerges from the data rather than exhaustive search, the method scales more readily to additional tasks.

Further work is needed to determine the optimal depth at which the shared encoder should branch into separate decoders and to quantify how task relationships evolve with network capacity. The main limitations are that most ablation studies used down-sampled images and that some benchmark comparisons involve models trained on additional external data; nevertheless, the core finding—that homoscedastic uncertainty provides a robust, automatic weighting mechanism—holds across both regimes and initialization choices.

University of CambridgeUniversity of Oxfordclassificationclassification modelsdeep learningmulti-task learningtask-dependent uncertaintymulti-task loss function
arXiv: 1705.07115yaringal/multi-task-learning-example
Cover for Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics

Abstract

Numerous deep learning applications benefit from multi-task learning with multiple regression and classification objectives. In this paper we make the observation that the performance of such systems is strongly dependent on the relative weighting between each task's loss. Tuning these weights by hand is a difficult and expensive process, making multi-task learning prohibitive in practice. We propose a principled approach to multi-task deep learning which weighs multiple loss functions by considering the homoscedastic uncertainty of each task. This allows us to simultaneously learn various quantities with different units or scales in both classification and regression settings. We demonstrate our model learning per-pixel depth regression, semantic and instance segmentation from a monocular input image. Perhaps surprisingly, we show our model can learn multi-task weightings and outperform separate models trained individually on each task.

Topics

classificationclassification modelsdeep learninggeometrymultiple regressionsemanticsmultitask learningonline learningconvolutional neural networks

Keywords

multi-task learningtask-dependent uncertaintymulti-task loss functionscene geometrysemantic segmentation taskdepth estimationscene understandinguncertainty estimates

Table of Contents

  • 1. Introduction
  • 2. Related Work
  • 3. Multi Task Learning with Homoscedastic Uncertainty
  • 3.1. Homoscedastic uncertainty as task-dependent uncertainty
  • 3.2. Multi-task likelihoods
  • 4. Scene Understanding Model
  • 5. Experiments
  • 5.1. Model Analysis
  • 6. Conclusions
  • A. Model Architecture Details
  • A.1. Optimisation
  • B. Further Analysis
  • C. Further Qualitative Results
  • D. Failure Examples

Knowls

  1. Knowl 1 — Multi-Task Loss Weighting via Homoscedastic Uncertainty

    model/method

    Multi-task learning simultaneously optimizes a shared network representation fW(x)f^W(x) with weights WW on input xx across multiple regression and classification objectives. Rather than relying on fixed or manually tuned linear scalar weights, homoscedastic (task-dependent) aleatoric uncertainty is treated as a learnable task-specific observation noise parameter σ\sigma.

    For a model with two continuous outputs y1y_1 and y2y_2, each modeled with a Gaussian likelihood N(yi;fW(x),σi2)\mathcal{N}(y_i; f^W(x), \sigma_i^2), maximizing the multi-task log-likelihood yields the minimization objective: L(W,σ1,σ2)=12σ12L1(W)+12σ22L2(W)+log⁡σ1+log⁡σ2\mathcal{L}(W, \sigma_1, \sigma_2) = \frac{1}{2\sigma_1^2}\mathcal{L}_1(W) + \frac{1}{2\sigma_2^2}\mathcal{L}_2(W) + \log \sigma_1 + \log \sigma_2 where L1(W)=∥y1−fW(x)∥2\mathcal{L}_1(W) = \|y_1 - f^W(x)\|^2 and L2(W)=∥y2−fW(x)∥2\mathcal{L}_2(W) = \|y_2 - f^W(x)\|^2 are the Euclidean losses for each task.

    For a joint model composed of a continuous regression output y1y_1 and a discrete classification output y2y_2 (modeled with a temperature-scaled softmax likelihood with scale σ2\sigma_2), the multi-task loss is formulated as: L(W,σ1,σ2)≈12σ12L1(W)+1σ22L2(W)+log⁡σ1+log⁡σ2\mathcal{L}(W, \sigma_1, \sigma_2) \approx \frac{1}{2\sigma_1^2}\mathcal{L}_1(W) + \frac{1}{\sigma_2^2}\mathcal{L}_2(W) + \log \sigma_1 + \log \sigma_2 where L2(W)=−log⁡Softmax(y2,fW(x))\mathcal{L}_2(W) = -\log \text{Softmax}(y_2, f^W(x)) is the standard discrete cross-entropy loss.

    In both cases, as the task observation noise σi\sigma_i increases, the relative weight of task loss Li(W)\mathcal{L}_i(W) decreases. The regularizing log⁡σi\log \sigma_i terms penalize excessively large noise values, preventing the loss weights from collapsing to zero.

  2. Knowl 2 — Log-Variance Parameterization for Homoscedastic Loss Optimization

    model/method

    Directly optimizing the noise variance parameters σ2\sigma^2 or standard deviations σ\sigma in homoscedastic multi-task loss functions can lead to numerical instability, including potential division by zero when σ→0\sigma \to 0. To stabilize optimization and allow unconstrained optimization over the real domain R\mathbb{R}, the network is parameterized to learn the log variance for each task ii: si:=log⁡σi2s_i := \log \sigma_i^2

    Applying the exponential transformation yields 1σi2=exp⁡(−si)\frac{1}{\sigma_i^2} = \exp(-s_i), which guarantees strictly positive variances σi2>0\sigma_i^2 > 0 for any real scalar si∈Rs_i \in \mathbb{R}.

    Under this parameterization, the joint loss function for a regression task with loss L1(W)\mathcal{L}_1(W) and a classification task with loss L2(W)\mathcal{L}_2(W) is: L(W,s1,s2)=12exp⁡(−s1)L1(W)+exp⁡(−s2)L2(W)+12s1+12s2\mathcal{L}(W, s_1, s_2) = \frac{1}{2} \exp(-s_1) \mathcal{L}_1(W) + \exp(-s_2) \mathcal{L}_2(W) + \frac{1}{2} s_1 + \frac{1}{2} s_2

  3. Knowl 3 — Instance Segmentation via Pixel Centroid Vector Voting and OPTICS Clustering

    model/method

    Instance segmentation is framed as a dense 2D offset regression task followed by density-based clustering in Hough parameter space. For every pixel coordinate cn=(un,vn)c_n = (u_n, v_n) that belongs to an instance class, the network predicts an instance vector x^n=(Δun,Δvn)\hat{x}_n = (\Delta u_n, \Delta v_n) pointing to the centroid ini_n of that object instance, such that in=cn+x^ni_n = c_n + \hat{x}_n.

    The regression objective is trained using an L1L_1 loss over all labeled instance pixels NIN_I in a mini-batch: LInstance=1∣NI∣∑n∈NI∥xn−x^n∥1\mathcal{L}_{\text{Instance}} = \frac{1}{|N_I|} \sum_{n \in N_I} \|x_n - \hat{x}_n\|_1 where xn=in−cnx_n = i_n - c_n is the ground-truth offset vector.

    At inference time, each pixel's predicted centroid location cn+x^nc_n + \hat{x}_n acts as a vote in coordinate space. The OPTICS density-based clustering algorithm is applied to group these votes into instance clusters without requiring an a priori estimate of the number of instances kk or assuming uniform instance scales/densities. Each pixel pnp_n is then assigned to the nearest estimated instance centroid cluster i^\hat{i}. This geometric voting mechanism allows the model to correctly identify and group disconnected masks belonging to the same instance when objects are partitioned by occlusions.

  4. Knowl 4 — Unified Multi-Task Scene Understanding Encoder-Decoder Architecture

    model/method

    The scene understanding architecture jointly predicts semantic segmentation, instance segmentation, and metric inverse depth from a single monocular RGB image using a shared encoder and task-specific decoders:

    1. Shared Feature Encoder: Utilizes a DeepLabV3 trunk with a ResNet-101 base network modified with dilated convolutions, reducing the feature map spatial resolution by a factor of 8 relative to the input image. An Atrous Spatial Pyramid Pooling (ASPP) module processes these features with four parallel convolutional layers (256 output channels each; dilation rates 1,12,24,361, 12, 24, 36; kernel sizes 1×1,3×3,3×3,3×31\times 1, 3\times 3, 3\times 3, 3\times 3) and a global average pooling branch projected to 256 channels with a 1×11\times 1 convolution. After batch normalization, these five branches are concatenated to yield a 1280-dimensional shared feature representation.

    2. Task-Specific Decoders: The shared feature representation branches into three separate decoders. Each decoder consists of a 3×33\times 3 convolutional layer producing 256 feature channels, followed by batch normalization, a non-linear activation, and a 1×11\times 1 convolution projecting to the required output channels (CC channels for CC semantic classes, 2 channels for 2D instance centroid vectors, and 1 channel for inverse depth). Bilinear upsampling scales the decoder outputs back to the original input resolution.

  5. Knowl 5 — Multi-Task Learning Performance Comparison on Tiny CityScapes

    data/table

    Performance comparison of single-task models, fixed loss weightings, manually tuned weights, and homoscedastic uncertainty weighting on the Tiny CityScapes validation set (downsampled to 128×256128 \times 256 resolution, trained for 50,000 iterations):

    Loss Task Weights Segmentation Instance Inverse Depth
    Seg. Inst. Depth IoU [%] Mean Error [px] Mean Error [px]
    Segmentation only 1 0 0 59.4% - -
    Instance only 0 1 0 - 4.61 -
    Depth only 0 0 1 - - 0.640
    Unweighted sum of losses 0.333 0.333 0.333 50.1% 3.79 0.592
    Approx. optimal weights 0.89 0.01 0.1 62.8% 3.61 0.549
    2 task uncertainty weighting ✓ ✓ 61.0% 3.42 -
    2 task uncertainty weighting ✓ ✓ 62.7% - 0.533
    2 task uncertainty weighting ✓ ✓ - 3.54 0.539
    3 task uncertainty weighting ✓ ✓ ✓ 63.4% 3.50 0.522

    Key empirical findings from this data include:

    • Naive uniform loss weighting (0.3330.333 for each task) severely harms semantic segmentation performance (IoU drops to 50.1% compared to 59.4% for the single-task baseline) due to differing loss scales.
    • Automatically learned homoscedastic uncertainty weighting across all 3 tasks achieves the highest performance on all three metrics (63.4% Segmentation IoU, 3.50 px Instance Error, 0.522 px Depth Error), outperforming all single-task models and manually grid-searched fixed weights.
    • Training on all 3 tasks jointly with uncertainty weighting outperforms every 2-task subset.

  6. Knowl 6 — CityScapes Full-Resolution Multi-Task Benchmark Results

    data/table

    Evaluation of the unified multi-task architecture on the CityScapes test dataset at full 1024×20481024 \times 2048 resolution across semantic segmentation, instance segmentation, and monocular disparity estimation:

    Method Semantic Segmentation Instance Segmentation Monocular Disparity
    IoU class iIoU class IoU cat iIoU cat AP AP 50% AP 100m AP 50m Mean Err [px] RMS Err [px]
    Multi-Task Learning 78.5 57.4 89.9 77.7 21.6 39.0 35.0 37.0 2.92 5.88
    Uhrig et al. 64.3 41.6 85.9 73.9 8.9 21.1 15.3 16.7 - -
    Mask R-CNN - - - - 26.2 49.9 37.6 40.1 - -
    Deep Watershed - - - - 19.4 35.3 31.4 36.8 - -
    R-CNN + MCG - - - - 4.6 12.9 7.7 10.3 - -
    DeepLab V3 81.3 60.9 91.6 81.7 - - - - - -
    PSPNet 81.2 59.6 91.2 79.2 - - - - - -
    Adelaide 71.6 51.7 87.3 74.1 - - - - - -

    The multi-task network is the first single model to perform semantic segmentation, instance segmentation, and depth estimation simultaneously. It substantially outperforms prior multi-task baselines performing segmentation and instance tasks (Uhrig et al. achieved 64.3% IoU class and 8.9% AP) while maintaining competitive performance with specialized single-task architectures.

  7. Knowl 7 — Softmax Likelihood Temperature Scaling for Classification Uncertainty

    theoretical result

    For classification tasks over CC classes with model output vector fW(x)f^W(x), homoscedastic uncertainty is incorporated by scaling the logits by temperature σ2>0\sigma^2 > 0 in a Boltzmann distribution: p(y=c∣fW(x),σ)=Softmax(1σ2fW(x))=exp⁡(1σ2fcW(x))∑c′exp⁡(1σ2fc′W(x))p(y = c \mid f^W(x), \sigma) = \text{Softmax}\left(\frac{1}{\sigma^2} f^W(x)\right) = \frac{\exp\left(\frac{1}{\sigma^2} f_c^W(x)\right)}{\sum_{c'} \exp\left(\frac{1}{\sigma^2} f_{c'}^W(x)\right)}

    The corresponding log-likelihood is: log⁡p(y=c∣fW(x),σ)=1σ2fcW(x)−log⁡∑c′exp⁡(1σ2fc′W(x))\log p(y = c \mid f^W(x), \sigma) = \frac{1}{\sigma^2} f_c^W(x) - \log \sum_{c'} \exp\left(\frac{1}{\sigma^2} f_{c'}^W(x)\right)

    To simplify the optimization objective and prevent complex second-order partition functions, the paper introduces the assumption: 1σ2∑c′exp⁡(1σ2fc′W(x))≈(∑c′exp⁡(fc′W(x)))1σ2\frac{1}{\sigma^2} \sum_{c'} \exp\left(\frac{1}{\sigma^2} f_{c'}^W(x)\right) \approx \left(\sum_{c'} \exp(f_{c'}^W(x))\right)^{\frac{1}{\sigma^2}} which holds with equality when σ→1\sigma \to 1. Under this simplification, the negative log-likelihood of the classification task reduces to: −log⁡p(y=c∣fW(x),σ)≈1σ2LCE(W)+log⁡σ-\log p(y = c \mid f^W(x), \sigma) \approx \frac{1}{\sigma^2} \mathcal{L}_{\text{CE}}(W) + \log \sigma where LCE(W)=−log⁡Softmax(y,fW(x))\mathcal{L}_{\text{CE}}(W) = -\log \text{Softmax}(y, f^W(x)) is the standard unscaled cross-entropy loss.

  8. Knowl 8 — Metric Inverse Depth Regression Formulation

    model/method

    Monocular depth estimation is formulated as predicting pixel-wise metric inverse depth d^n\hat{d}_n rather than raw Euclidean depth znz_n, because inverse depth representation allows mapping points at infinite distance to finite values (dn=0d_n = 0).

    The depth regression decoder is trained using an L1L_1 loss over the set of valid labeled pixels NDN_D: LDepth=1∣ND∣∑n∈ND∣dn−d^n∣\mathcal{L}_{\text{Depth}} = \frac{1}{|N_D|} \sum_{n \in N_D} |d_n - \hat{d}_n|

    For outdoor scene understanding, pixels with semantic ground truth label 'sky' are assigned an inverse depth target of zero (dn=0d_n = 0). Pixels without valid ground truth depth or stereo measurements are masked out and ignored in the loss computation.

  9. Knowl 9 — Convergence Dynamics and Initialization Robustness of Homoscedastic Task Weights

    empirical result

    Homoscedastic uncertainty optimization exhibits rapid convergence and robustness to hyperparameter initialization:

    • Initialization Invariance: When initial values for log⁡σ2\log \sigma^2 are varied across the range [−2.0,5.0][-2.0, 5.0], the homoscedastic noise parameters converge to identical optima in approximately 100 iterations, despite the full deep network requiring over 30,000 iterations to train.
    • Dynamic Weight Evolution: Task uncertainties decrease as training progresses and the network improves at each task. On CityScapes, task variances (σseg2,σinst2,σdepth2)(\sigma_{\text{seg}}^2, \sigma_{\text{inst}}^2, \sigma_{\text{depth}}^2) shift from (0.60,62.5,13.5)(0.60, 62.5, 13.5) at iteration 500 to (0.075,3.25,20.4)(0.075, 3.25, 20.4) at convergence. This corresponds to an effective loss weighting ratio shift from 23:0.22:123 : 0.22 : 1 early in training to 43:0.16:143 : 0.16 : 1 at the end of training.
    • Learning Rate Interaction: Because the estimated uncertainties decrease over time, the effective weight on each task's loss increases throughout training. To prevent training instability from expanding effective gradient magnitudes, the optimization uses polynomial learning rate decay: η=η0(1−itermax_iter)0.9\eta = \eta_0 \left(1 - \frac{\text{iter}}{\text{max\_iter}}\right)^{0.9}.

  10. Knowl 10 — Cross-Modal Error Coupling in Multi-Task Shared Representations

    limitation

    Because semantic segmentation, instance segmentation, and depth estimation decoders are all conditioned on a shared encoder trunk, prediction errors often exhibit strong cross-modal coupling.

    Specifically, when the network misclassifies image content in the semantic branch (e.g., classifying a building window reflection as a person, or confusing road surface with footpath), the depth regression branch simultaneously predicts geometric contours matching the incorrect semantic class (e.g., predicting human depth profile and distance for the reflection). Similarly, severe occlusions or objects that exceed the receptive field of the shared feature encoder lead to concurrent breakdown in semantic masks, instance vector regression, and depth estimation.

Coverage note — None was omitted; all core contributions, theoretical derivations, architectural components, benchmark evaluations, and analytical findings have been captured.

References

  1. 1.P. Agrawal, J. Carreira, and J. Malik. Learning to see by moving. In Proceedings of the IEEE International Conference on Computer Vision, pages 37–45, 2015. 2
  2. 2.M. Ankerst, M. M. Breunig, H.-P. Kriegel, and J. Sander. Optics: ordering points to identify the clustering structure. In ACM Sigmod Record, volume 28, pages 49–60. ACM, 1999. 6
  3. 3.V. Badrinarayanan, A. Kendall, and R. Cipolla. Segnet: A deep convolutional encoder-decoder architecture for scene segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017. 1, 5
  4. 4.M. Bai and R. Urtasun. Deep watershed transform for instance segmentation. arXiv preprint arXiv:1611.08303, 2016. 1, 6, 8
  5. 5.J. Baxter et al. A model of inductive bias learning. J. Artif. Intell. Res.(JAIR), 12(149-198):3, 2000. 2
  6. 6.S. R. Bulò, L. Porzi, and P. Kontschieder. In-place activated batchnorm for memory-optimized training of dnns. arXiv preprint arXiv:1712.02616, 2017.
  7. 7.R. Caruana. Multitask learning. In Learning to learn, pages 95–133. Springer, 1998. 1, 2
  8. 8.L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. In ICLR, 2015. 1
  9. 9.L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. arXiv preprint arXiv:1606.00915, 2016.
  10. 10.L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017. 5, 8, 11
  11. 11.R. Collobert and J. Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM, 2008. 1, 2
  12. 12.D. Comaniciu and P. Meer. Mean shift: A robust approach toward feature space analysis. IEEE Transactions on pattern analysis and machine intelligence, 24(5):603–619, 2002. 6
  13. 13.M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, 2016. 6, 8
  14. 14.J. Dai, K. He, and J. Sun. Instance-aware semantic segmentation via multi-task network cascades. In In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, 2016. 1
  15. 15.D. Eigen and R. Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In Proceedings of the IEEE International Conference on Computer Vision, pages 2650–2658, 2015. 1, 2, 3
  16. 16.R. Garg and I. Reid. Unsupervised cnn for single view depth estimation: Geometry to the rescue. Computer Vision–ECCV 2016, pages 740–756, 2016. 1
  17. 17.R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pages 580–587, 2014. 1
  18. 18.B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik. Hypercolumns for object segmentation and fine-grained localization. In In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pages 447–456. IEEE, 2014. 1
  19. 19.K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. arXiv preprint arXiv:1703.06870, 2017. 8
  20. 20.K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, 2016. 5, 11
  21. 21.H. Hirschmuller. Accurate and efficient stereo processing by semi-global matching and mutual information. In In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, volume 2, pages 807–814. IEEE, 2005. 8
  22. 22.H. Hirschmuller. Stereo processing by semiglobal matching and mutual information. IEEE Transactions on pattern analysis and machine intelligence, 30(2):328–341, 2008. 6
  23. 23.J.-T. Huang, J. Li, D. Yu, L. Deng, and Y. Gong. Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages 7304–7308. IEEE, 2013. 1
  24. 24.A. Kendall and Y. Gal. What uncertainties do we need in bayesian deep learning for computer vision? arXiv preprint arXiv:1703.04977, 2017. 4
  25. 25.A. Kendall, M. Grimes, and R. Cipolla. Convolutional networks for real-time 6-dof camera relocalization. In Proceedings of the International Conference on Computer Vision (ICCV), 2015. 2, 3
  26. 26.J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, page 201611835, 2017. 2
  27. 27.I. Kokkinos. Ubernet: Training auniversal'convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. arXiv preprint arXiv:1609.02132, 2016. 1, 2, 3
  28. 28.B. Leibe, A. Leonardis, and B. Schiele. Robust object detection with interleaved categorization and segmentation. International Journal of Computer Vision (IJCV), 77(1-3):259–289, 2008. 6
  29. 29.X. Liang, Y. Wei, X. Shen, J. Yang, L. Lin, and S. Yan. Proposal-free network for instance-level object segmentation. arXiv preprint arXiv:1509.02636, 2015. 6
  30. 30.Y. Liao, S. Kodagoda, Y. Wang, L. Shi, and Y. Liu. Understand scene categories by objects: A semantic regularized scene classifier using convolutional neural networks. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 2318–2325. IEEE, 2016. 2, 3
  31. 31.G. Lin, C. Shen, I. Reid, et al. Efficient piecewise training of deep structured models for semantic segmentation. arXiv preprint arXiv:1504.01013, 2015. 8
  32. 32.J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, 2015. 1
  33. 33.J. MacQueen et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1, pages 281–297. Oakland, CA, USA., 1967. 6
  34. 34.I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3994–4003, 2016. 2
  35. 35.J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng. Multimodal deep learning. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 689–696, 2011. 2
  36. 36.M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pages 1717–1724. IEEE, 2014. 2
  37. 37.P. O. Pinheiro, R. Collobert, and P. Dollar. Learning to segment object candidates. In Advances in Neural Information Processing Systems, pages 1990–1998, 2015. 1
  38. 38.P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. International Conference on Learning Representations (ICLR), 2014. 1, 2, 3
  39. 39.M. Teichmann, M. Weber, M. Zoellner, R. Cipolla, and R. Urtasun. Multinet: Real-time joint semantic reasoning for autonomous driving. arXiv preprint arXiv:1612.07695, 2016. 2, 3
  40. 40.S. Thrun. Is learning the n-th thing any easier than learning the first? In Advances in neural information processing systems, pages 640–646. MORGAN KAUFMANN PUBLISHERS, 1996. 2
  41. 41.J. Uhrig, M. Cordts, U. Franke, and T. Brox. Pixel-level encoding and depth layering for instance-level semantic labeling. arXiv preprint arXiv:1604.05096, 2016. 2, 3, 8
  42. 42.F. Yu and V. Koltun. Multi-scale context aggregation by dilated convolutions. In ICLR, 2016. 1
  43. 43.S. Zagoruyko and N. Komodakis. Wide residual networks. In E. R. H. Richard C. Wilson and W. A. P. Smith, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 87.1–87.12. BMVA Press, September 2016.
  44. 44.H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. arXiv preprint arXiv:1612.01105, 2016. 8
  45. 45.S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, and P. Torr. Conditional random fields as recurrent neural networks. In International Conference on Computer Vision (ICCV), 2015. 1

Citation

MLA
Kendall, A., et al. “Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics”. arXiv, 2017, http://arxiv.org/abs/1705.07115v3.
APA
Kendall, A., Gal, Y., & Cipolla, R. (2017). Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics. arXiv. http://arxiv.org/abs/1705.07115v3
Chicago
Kendall, A., Y. Gal, and R. Cipolla. 2017. “Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics”. arXiv. http://arxiv.org/abs/1705.07115v3.
Harvard
Kendall, A., Gal, Y. and Cipolla, R. (2017) “Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1705.07115v3.
Vancouver
1. Kendall A, Gal Y, Cipolla R (2017) Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics. arXiv

BibTeX

@article{kendall2017multi,
  title = {Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics},
  author = {Kendall, Alex and Gal, Yarin and Cipolla, Roberto},
  year = {2017},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1705.07115v3},
  eprint = {1705.07115}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE