Back

GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks

Zhao ChenVijay BadrinarayananChen-Yu LeeAndrew Rabinovich

article2017ICML2,071 citations

Proposes GradNorm, an adaptive loss-balancing method that dynamically scales gradient magnitudes during training, eliminating expensive loss-weight grid searches while improving multitask performance and reducing overfitting across diverse architectures.

classificationclassification methodsmultitask learningGradNormLossGradient
arXiv: 1711.02257
  • Paper: Multitask Learning, RICH CARUANA (1997). Provides foundational principles and architectures for multi-task learning through shared representations, establishing the baseline framework that GradNorm dynamically optimizes.
  • Paper: An Overview of Multi-Task Learning in Deep Neural Networks, Sebastian Ruder (2017). Surveys the mechanics and core challenges of hard and soft parameter sharing in deep multi-task networks, providing direct context for the multi-task optimization bottleneck GradNorm targets.
  • Paper: A Survey on Multi-Task Learning, Yu Zhang et al. (2017). Offers a comprehensive taxonomy of multi-task learning algorithms and feature-sharing strategies, clarifying the broader problem space of joint task optimization.
  • Paper: An overview of gradient descent optimization algorithms, Sebastian Ruder (2016). Reviews fundamental gradient descent optimization techniques and adaptive learning rate mechanics that underlie gradient-based balancing algorithms.
Cover for GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks

Abstract

Deep multitask networks, in which one neural network produces multiple predictive outputs, can offer better speed and performance than their single-task counterparts but are challenging to train properly. We present a gradient normalization (GradNorm) algorithm that automatically balances training in deep multitask models by dynamically tuning gradient magnitudes. We show that for various network architectures, for both regression and classification tasks, and on both synthetic and real datasets, GradNorm improves accuracy and reduces overfitting across multiple tasks when compared to single-task networks, static baselines, and other adaptive multitask loss balancing techniques. GradNorm also matches or surpasses the performance of exhaustive grid search methods, despite only involving a single asymmetry hyperparameter α\alpha. Thus, what was once a tedious search process that incurred exponentially more compute for each task added can now be accomplished within a few training runs, irrespective of the number of tasks. Ultimately, we will demonstrate that gradient manipulation affords us great control over the training dynamics of multitask networks and may be one of the keys to unlocking the potential of multitask learning.

Topics

classificationclassification methodsmultitask learningnetwork architectureneural networkssearch processreal data setsregularization parametersgrid nodeadaptive algorithms

Keywords

GradNormLossGradientMultitaskNetworksTasksTaskDeepDeep Multitask NetworksWeights

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 The GradNorm Algorithm
  • 3.1 Definitions and Preliminaries
  • 3.2 Balancing Gradients with GradNorm
  • 4 A Toy Example
  • 5 Application to a Large Real-World Dataset
  • 5.1 Model and General Training Characteristics
  • 5.2 Main Results on NYUv2
  • 5.3 Gradient Normalization Finds Optimal Grid-Search Weights in One Pass
  • 5.4 Effects of tuning the asymmetry α\alpha
  • 5.5 Qualitative Results
  • 6 Conclusions
  • References
  • 7 GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks: Supplementary Materials
  • 7.1 Performance Gains Versus α\alpha
  • 7.2 Additional Experiments on a Multitask Facial Landmark Dataset

Knowls

  1. Knowl 1 — GradNorm Multitask Loss Balancing Formulation

    model/method

    Gradient Normalization (GradNorm) is an adaptive loss-balancing technique for deep multitask networks that dynamically adjusts task weights wi(t)w_i(t) at each training step tt to balance gradient magnitudes and equalize task training rates.

    Let the global multitask loss be defined as a weighted sum over TT individual task losses Li(t)L_i(t):

    L(t)=∑i=1Twi(t)Li(t)L(t) = \sum_{i=1}^T w_i(t) L_i(t)

    Let W⊂WW \subset \mathcal{W} denote a selected subset of shared model parameters, chosen as the weights of the final shared layer to minimize computation overhead. GradNorm tracks the following quantities at time tt:

    1. The L2L_2 norm of the gradient of the weighted task loss wi(t)Li(t)w_i(t) L_i(t) with respect to WW:

    GW(i)(t)=∥∇W(wi(t)Li(t))∥2G_W^{(i)}(t) = \|\nabla_W (w_i(t) L_i(t))\|_2

    1. The average gradient norm across all TT tasks:

    GˉW(t)=Etask[GW(i)(t)]=1T∑i=1TGW(i)(t)\bar{G}_W(t) = \mathbb{E}_{\text{task}}[G_W^{(i)}(t)] = \frac{1}{T} \sum_{i=1}^T G_W^{(i)}(t)

    1. The loss ratio (inverse training progress) of task ii relative to its initial loss Li(0)L_i(0):

    L~i(t)=Li(t)Li(0)\tilde{L}_i(t) = \frac{L_i(t)}{L_i(0)}

    (When initial loss depends heavily on random initialization, theoretical initial values such as Li(0)=log⁡(C)L_i(0) = \log(C) for a CC-class cross-entropy loss may be used instead).

    1. The relative inverse training rate ri(t)r_i(t) of task ii:

    ri(t)=L~i(t)Etask[L~i(t)]=L~i(t)1T∑j=1TL~j(t)r_i(t) = \frac{\tilde{L}_i(t)}{\mathbb{E}_{\text{task}}[\tilde{L}_i(t)]} = \frac{\tilde{L}_i(t)}{\frac{1}{T}\sum_{j=1}^T \tilde{L}_j(t)}

    A higher ri(t)r_i(t) signifies that task ii is training more slowly than the average task.

    GradNorm targets a desired gradient norm GˉW(t)×[ri(t)]α\bar{G}_W(t) \times [r_i(t)]^\alpha for each task ii, where α≥0\alpha \ge 0 is an asymmetry hyperparameter. The task weights wi(t)w_i(t) are updated by minimizing the L1L_1 gradient loss:

    Lgrad(t;w1(t),…,wT(t))=∑i=1T∣GW(i)(t)−GˉW(t)×[ri(t)]α∣L_{\text{grad}}(t; w_1(t), \dots, w_T(t)) = \sum_{i=1}^T \left| G_W^{(i)}(t) - \bar{G}_W(t) \times [r_i(t)]^\alpha \right|

    When calculating ∇wiLgrad\nabla_{w_i} L_{\text{grad}}, the target values GˉW(t)×[ri(t)]α\bar{G}_W(t) \times [r_i(t)]^\alpha are treated as fixed constants to prevent loss weights from decaying to zero. After updating wi(t)w_i(t) via gradient descent on LgradL_{\text{grad}}, the weights are renormalized so that ∑i=1Twi(t)=T\sum_{i=1}^T w_i(t) = T, decoupling loss weight optimization from the global learning rate.

  2. Knowl 2 — GradNorm Training Algorithm

    algorithm

    GradNorm dynamically updates task weights wi(t)w_i(t) via gradient descent on the gradient norm loss LgradL_{\text{grad}} while training the network parameters W\mathcal{W} on the multitask loss L(t)L(t).

    Input: Training data batches xix_i, number of tasks TT, maximum training steps tmax⁡t_{\max}, asymmetry hyperparameter α>0\alpha > 0, task weight learning rate ηw\eta_w, model learning rate ηW\eta_W, shared layer weights W⊂WW \subset \mathcal{W}
    Initialize loss weights wi(0)=1w_i(0) = 1 for all i∈{1,…,T}i \in \{1, \dots, T\}
    Initialize network parameters W\mathcal{W}
    for t=0t = 0 to tmax⁡−1t_{\max} - 1 do
        Forward pass on batch xtx_t to compute task losses Li(t)L_i(t) for all i∈{1,…,T}i \in \{1, \dots, T\}
        Compute global multitask loss L(t)=∑i=1Twi(t)Li(t)L(t) = \sum_{i=1}^T w_i(t) L_i(t)
        for i=1i = 1 to TT do
            Compute gradient norm GW(i)(t)=∥∇W(wi(t)Li(t))∥2G_W^{(i)}(t) = \|\nabla_W (w_i(t) L_i(t))\|_2
            Compute loss ratio L~i(t)=Li(t)/Li(0)\tilde{L}_i(t) = L_i(t) / L_i(0)
        end for
        Compute mean gradient norm GˉW(t)=1T∑i=1TGW(i)(t)\bar{G}_W(t) = \frac{1}{T} \sum_{i=1}^T G_W^{(i)}(t)
        Compute relative inverse training rates ri(t)=L~i(t)/(1T∑j=1TL~j(t))r_i(t) = \tilde{L}_i(t) / (\frac{1}{T} \sum_{j=1}^T \tilde{L}_j(t)) for all ii
        Compute gradient loss Lgrad=∑i=1T∣GW(i)(t)−GˉW(t)×[ri(t)]α∣L_{\text{grad}} = \sum_{i=1}^T |G_W^{(i)}(t) - \bar{G}_W(t) \times [r_i(t)]^\alpha|
        Compute GradNorm gradients ∇wiLgrad\nabla_{w_i} L_{\text{grad}} treating target values GˉW(t)×[ri(t)]α\bar{G}_W(t) \times [r_i(t)]^\alpha as constants
        Compute network weight gradients ∇WL(t)\nabla_\mathcal{W} L(t)
        Update loss weights wi(t+1)=wi(t)−ηw∇wiLgradw_i(t + 1) = w_i(t) - \eta_w \nabla_{w_i} L_{\text{grad}}
        Renormalize weights wi(t+1)=T⋅wi(t+1)∑j=1Twj(t+1)w_i(t + 1) = T \cdot \frac{w_i(t + 1)}{\sum_{j=1}^T w_j(t + 1)}
        Update network parameters W(t+1)=W(t)−ηW∇WL(t)\mathcal{W}(t + 1) = \mathcal{W}(t) - \eta_W \nabla_\mathcal{W} L(t)
    end for

  3. Knowl 3 — Role of the Asymmetry Hyperparameter in GradNorm

    model/method

    The asymmetry hyperparameter α≥0\alpha \ge 0 in GradNorm controls the strength of the restoring force that pulls tasks toward a common training pace:

    • When α=0\alpha = 0, the target gradient norm for each task ii simplifies to GˉW(t)\bar{G}_W(t), forcing all tasks' backpropagated gradient norms at the chosen layer WW to be equal regardless of their relative training speeds.
    • When α>0\alpha > 0, the target is modulated by [ri(t)]α[r_i(t)]^\alpha, where ri(t)=L~i(t)/E[L~i(t)]r_i(t) = \tilde{L}_i(t) / \mathbb{E}[\tilde{L}_i(t)] represents the relative inverse training speed. Tasks that train more slowly than average (ri(t)>1r_i(t) > 1) receive higher target gradient norms, prompting an increase in wi(t)w_i(t). Conversely, tasks training quickly (ri(t)<1r_i(t) < 1) have their gradient norms and weights suppressed.
    • Increasing α\alpha spreads the weights wi(t)w_i(t) further apart, which acts as strong regularization by suppressing weights of tasks prone to fast convergence or overfitting. Symmetric or uniformly scaled synthetic tasks require low α\alpha (e.g., α=0.12\alpha = 0.12), whereas complex real-world vision datasets with disparate task difficulties benefit from higher values (e.g., α=1.5\alpha = 1.5 on NYUv2).

  4. Knowl 4 — Synthetic Multi-Scale Regression Dynamics: GradNorm vs. Uncertainty Weighting

    empirical result

    To evaluate loss balancing across different output loss scales, TT regression tasks were evaluated on synthetic data generated from:

    fi(x)=σitanh⁡((B+ϵi)x)f_i(x) = \sigma_i \tanh((B + \epsilon_i)x)

    where inputs x∈R250x \in \mathbb{R}^{250}, outputs fi(x)∈R100f_i(x) \in \mathbb{R}^{100}, Bjk∼N(0,10)B_{jk} \sim \mathcal{N}(0, 10), ϵi,jk∼N(0,3.5)\epsilon_{i,jk} \sim \mathcal{N}(0, 3.5), and σi\sigma_i are fixed scalars setting task output scales. Networks used a 4-layer fully connected trunk (100 neurons per layer, ReLU) with an affine output layer for each task, using α=0.12\alpha = 0.12.

    Performance was measured using the task-normalized test loss ∑i=1TLi(t)/Li(0)\sum_{i=1}^T L_i(t)/L_i(0):

    1. Equal Weighting (wi=1w_i = 1): For T=2T=2 with (σ0,σ1)=(1.0,100.0)(\sigma_0, \sigma_1) = (1.0, 100.0), task 1 dominated backpropagated gradients, suppressing learning in task 0 and causing high test loss.
    2. Uncertainty Weighting: Weighting losses inversely by task loss/variance (wi(t)∼1/Li(t)w_i(t) \sim 1/L_i(t)) without constraints caused wi(t)w_i(t) to grow unchecked as losses dropped. This escalated the effective global learning rate and caused training to deteriorate.
    3. GradNorm: Dynamically elevated w0(t)w_0(t) relative to w1(t)w_1(t) to equalize gradients while maintaining ∑iwi(t)=T\sum_i w_i(t) = T. On both T=2T=2 and T=10T=10 (where σi\sigma_i were sampled from a wide normal distribution), GradNorm produced stable, convergent task weights and significantly lower task-normalized test loss than equal weighting and uncertainty weighting.

  5. Knowl 5 — Performance on the NYUv2 Multitask Benchmark

    data/table

    GradNorm was evaluated on two variants of the NYUv2 indoor scene dataset: NYUv2+seg (795 train, 654 test images; depth RMS error, 13-class semantic segmentation 100−IoU100-\text{IoU} error, surface normal 1−∣cos⁡∣1-|\cos| error) and NYUv2+kpts (90,000 augmented images; depth RMS error, 48 room keypoint heatmap error percentage, surface normal error). Models used either a VGG16 SegNet (29M parameters) or a thin ResNet-50 FCN (15M parameters) sharing all parameters except the final prediction layer.

    Model and Weighting Method Depth RMS Err. (m) Seg. Err. (100-IoU) Normals Err. (1-|cos|)
    VGG Backbone (NYUv2+seg)
    Depth Only 1.038 - -
    Seg. Only - 70.0 -
    Normals Only - - 0.169
    Equal Weights 0.944 70.1 0.192
    GradNorm Static 0.939 67.5 0.171
    GradNorm α=1.5\alpha = 1.5 0.925 67.8 0.174
    Model and Weighting Method Depth RMS Err. (m) Kpt. Err. (%) Normals Err. (1-|cos|)
    ResNet Backbone (NYUv2+kpts)
    Depth Only 0.725 - -
    Kpt Only - 7.90 -
    Normals Only - - 0.155
    Equal Weights 0.697 7.80 0.172
    Kendall et al. (2017) 0.702 7.96 0.182
    GradNorm Static 0.695 7.63 0.156
    GradNorm α=1.5\alpha = 1.5 0.663 7.32 0.155
    VGG Backbone (NYUv2+kpts)
    Depth Only 0.689 - -
    Keypoint Only - 8.39 -
    Normals Only - - 0.142
    Equal Weights 0.658 8.39 0.155
    Kendall et al. (2017) 0.649 8.00 0.158
    GradNorm Static 0.638 7.69 0.137
    GradNorm α=1.5\alpha = 1.5 0.629 7.73 0.139

    On both benchmarks and both backbone architectures, dynamic GradNorm (α=1.5\alpha = 1.5) improved performance over equal weights and uncertainty weighting across all tasks, matching or outperforming single-task networks. On NYUv2+kpts, GradNorm improved test depth error by ∼5%\sim 5\% while converging to a higher training loss by suppressing wdepth(t)<0.10w_{\text{depth}}(t) < 0.10, demonstrating a strong regularizing effect.

  6. Knowl 6 — Optimal Static Task Weight Extraction via Time-Averaged GradNorm Weights

    empirical result

    GradNorm can find high-performing static multitask loss weights in a single training pass by calculating the time-averaged weights Et[wi(t)]\mathbb{E}_t[w_i(t)].

    In experiments on NYUv2+kpts with a VGG SegNet backbone:

    1. 100 networks were trained from scratch for 15,000 steps using static task weights wistaticw_i^{\text{static}} sampled uniformly at random and normalized to ∑i=13wistatic=3\sum_{i=1}^3 w_i^{\text{static}} = 3.
    2. The performance of these grid-searched static networks showed a strong negative correlation with the L2L_2 distance between wistaticw_i^{\text{static}} and the time-averaged GradNorm weights Et[wi(t)]\mathbb{E}_t[w_i(t)] from an α=1.5\alpha = 1.5 run.
    3. Networks with static weights at an L2L_2 distance of ∼3\sim 3 from Et[wi(t)]\mathbb{E}_t[w_i(t)] incurred almost double the error per task compared to the GradNorm network.
    4. Even after training 100 random networks, grid search failed to discover weight combinations that matched the performance achieved by GradNorm in a single training run, confirming that GradNorm's dynamic trajectories converge around the global optimum for static loss weighting.

  7. Knowl 7 — Performance on the Multi-Task Facial Landmark (MTFL) Dataset

    data/table

    The Multi-Task Facial Landmark (MTFL) dataset consists of ~13,000 face images (10,000 train, 3,000 test) labeled with five facial landmark coordinates (left eye, right eye, nose, left mouth corner, right mouth corner) and four attribute classification labels (gender, smiling, glasses, head pose). Models were evaluated across input resolutions of 40×4040 \times 40 and 160×160160 \times 160 using keypoint error (mean L2L_2 error normalized by inter-ocular distance), landmark failure rate (percentage of images with >10%>10\% keypoint error), and classification error percentages.

    Method Resolution Keypoint Err. (%) Failure Rate (%) Gender Err. (%) Smiles Err. (%) Glasses Err. (%) Pose Err. (%)
    Equal Weights 40x40 8.3 27.4 20.3 19.2 8.1 38.9
    Zhang et al. (2014) 40x40 8.2 25.0 - - - -
    Kendall et al. (2017) 40x40 8.3 27.2 20.7 18.5 8.1 38.9
    GradNorm α=0.3\alpha = 0.3 40x40 8.0 25.0 17.3 16.9 8.1 38.9
    Equal Weights 160x160 6.8 15.2 18.6 17.4 8.1 38.9
    Kendall et al. (2017) 160x160 7.2 18.3 38.1 18.4 8.1 38.9
    GradNorm α=0.2\alpha = 0.2 160x160 6.5 14.3 14.4 15.4 8.1 38.9

    GradNorm dynamically drove the keypoint loss weight wkpt(t)w_{\text{kpt}}(t) down to ≤0.01\le 0.01, reallocating gradient capacity to undertrained classification tasks. This significantly reduced gender and smiles classification errors while maintaining or improving keypoint regression accuracy. In contrast, uncertainty weighting (Kendall et al., 2017) increased the keypoint loss weight relative to classification, leading to degraded classification performance (e.g., 38.1%38.1\% gender error at 160×160160 \times 160).

  8. Knowl 8 — GradNorm Limitation on Stuck or Degenerate Task Objectives

    limitation

    GradNorm is vulnerable to pathological or degenerate tasks that become stuck during training, such as classifiers that quickly collapse to the majority class (e.g., glasses and pose classification on MTFL) and fail to make further loss progress.

    Because the task loss Li(t)L_i(t) does not decrease, the task's loss ratio L~i(t)=Li(t)/Li(0)\tilde{L}_i(t) = L_i(t)/L_i(0) remains near 1.01.0, while the loss ratios of improving tasks decrease toward 00. Consequently, the relative inverse training rate ri(t)r_i(t) grows disproportionately large, causing GradNorm to continually increase wi(t)w_i(t) in an attempt to accelerate training on the stagnant task, without yielding any learning improvement.

Coverage note — None was omitted; all contributed algorithms, mathematical formulations, synthetic experiments, real-world benchmark evaluations (NYUv2 and MTFL), hyperparameter dynamics, and stated limitations are fully covered.

References

  1. 1.Badrinarayanan, V., Kendall, A., and Cipolla, R. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. arXiv preprint arXiv:1511.00561, 2015.
  2. 2.Bakker, B. and Heskes, T. Task clustering and gating for bayesian multitask learning. Journal of Machine Learning Research, 4(May):83–99, 2003.
  3. 3.Bilen, H. and Vedaldi, A. Universal representations: The missing link between faces, text, planktons, and cat breeds. arXiv preprint arXiv:1701.07275, 2017.
  4. 4.Caruana, R. Multitask learning. In Learning to learn, pp. 95–133. Springer, 1998.
  5. 5.Collobert, R. and Weston, J. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pp. 160–167. ACM, 2008.
  6. 6.Eigen, D. and Fergus, R. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2650–2658, 2015.
  7. 7.Graves, A., Bellemare, M. G., Menick, J., Munos, R., and Kavukcuoglu, K. Automated curriculum learning for neural networks. arXiv preprint arXiv:1704.03003, 2017.
  8. 8.Hashimoto, K., Xiong, C., Tsuruoka, Y., and Socher, R. A joint many-task model: Growing a neural network for multiple nlp tasks. arXiv preprint arXiv:1611.01587, 2016.
  9. 9.He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  10. 10.He, K., Gkioxari, G., Dollár, P., and Girshick, R. Mask r-cnn. arXiv preprint arXiv:1703.06870, 2017.
  11. 11.Huang, W., Song, G., Hong, H., and Xie, K. Deep architecture for traffic flow prediction: deep belief networks with multitask learning. IEEE Transactions on Intelligent Transportation Systems, 15(5):2191–2201, 2014.
  12. 12.Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448–456, 2015.
  13. 13.Jacob, L., Vert, J.-p., and Bach, F. R. Clustered multi-task learning: A convex formulation. In Advances in neural information processing systems, pp. 745–752, 2009.
  14. 14.Kang, Z., Grauman, K., and Sha, F. Learning with whom to share in multi-task feature learning. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 521–528, 2011.
  15. 15.Kendall, A., Gal, Y., and Cipolla, R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. arXiv preprint arXiv:1705.07115, 2017.
  16. 16.Kokkinos, I. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. arXiv preprint arXiv:1609.02132, 2016.
  17. 17.Lee, C.-Y., Badrinarayanan, V., Malisiewicz, T., and Rabinovich, A. Roomnet: End-to-end room layout estimation. arXiv preprint arXiv:1703.06241, 2017.
  18. 18.Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440, 2015.
  19. 19.Long, M. and Wang, J. Learning multiple tasks with deep relationship networks. arXiv preprint arXiv:1506.02117, 2015.
  20. 20.Lu, Y., Kumar, A., Zhai, S., Cheng, Y., Javidi, T., and Feris, R. Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification. arXiv preprint arXiv:1611.05377, 2016.
  21. 21.Misra, I., Shrivastava, A., Gupta, A., and Hebert, M. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994–4003, 2016.
  22. 22.Nathan Silberman, Derek Hoiem, P. K. and Fergus, R. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
  23. 23.Redmon, J. and Farhadi, A. Yolo9000: better, faster, stronger. arXiv preprint arXiv:1612.08242, 2016.
  24. 24.Seltzer, M. L. and Droppo, J. Multi-task learning in deep neural networks for improved phoneme recognition. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pp. 6965–6969. IEEE, 2013.
  25. 25.Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  26. 26.Søgaard, A. and Goldberg, Y. Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, volume 2, pp. 231–235, 2016.
  27. 27.Teichmann, M., Weber, M., Zoellner, M., Cipolla, R., and Urtasun, R. Multinet: Real-time joint semantic reasoning for autonomous driving. arXiv preprint arXiv:1612.07695, 2016.
  28. 28.Warde-Farley, D., Rabinovich, A., and Anguelov, D. Self-informed neural network structure learning. arXiv preprint arXiv:1412.6563, 2014.
  29. 29.Wu, Z., Valentini-Botinhao, C., Watts, O., and King, S. Deep neural networks employing multi-task learning and stacked bottleneck features for speech synthesis. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, pp. 4460–4464. IEEE, 2015.
  30. 30.Zhang, Z., Luo, P., Loy, C. C., and Tang, X. Facial landmark detection by deep multi-task learning. In European Conference on Computer Vision, pp. 94–108. Springer, 2014.

Citation

MLA
Chen, Z., et al. “GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks”. Proceedings of the 35th International Conference on Machine Learning (2018), 793-802, 2017, http://arxiv.org/abs/1711.02257v4.
APA
Chen, Z., Badrinarayanan, V., Lee, C.-Y., & Rabinovich, A. (2017). GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks. Proceedings of the 35th International Conference on Machine Learning (2018), 793-802. http://arxiv.org/abs/1711.02257v4
Chicago
Chen, Z., V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich. 2017. “GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks”. Proceedings of the 35th International Conference on Machine Learning (2018), 793-802. http://arxiv.org/abs/1711.02257v4.
Harvard
Chen, Z. et al. (2017) “GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks”, Proceedings of the 35th International Conference on Machine Learning (2018), 793-802 [Preprint]. Available at: http://arxiv.org/abs/1711.02257v4.
Vancouver
1. Chen Z, Badrinarayanan V, Lee C-Y, Rabinovich A (2017) GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks. Proceedings of the 35th International Conference on Machine Learning (2018), 793-802

BibTeX

@article{chen2017gradnorm,
  title = {GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks},
  author = {Chen, Zhao and Badrinarayanan, Vijay and Lee, Chen-Yu and Rabinovich, Andrew},
  year = {2017},
  journal = {Proceedings of the 35th International Conference on Machine Learning (2018), 793-802},
  url = {http://arxiv.org/abs/1711.02257v4},
  eprint = {1711.02257}
}
Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/