Back

CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation

Yue WangWeishi WangShafiq JotySteven C. H. Hoi

article2021EMNLP2,485 citations

Presents CodeT5, a unified encoder-decoder Transformer that leverages identifier-aware pre-training and bimodal generation tasks to achieve state-of-the-art performance across both code understanding and synthesis benchmarks.

Listen

The article addresses challenges in applying pre-trained language models to programming tasks, where existing approaches often use encoder-only or decoder-only architectures that underperform on generation or understanding tasks respectively, and overlook code-specific features such as developer-assigned identifiers that carry semantic information.

The article set out to develop and evaluate a unified encoder-decoder model that supports both code understanding and generation while incorporating token-type information from code.

The authors built CodeT5 on the T5 architecture and pre-trained it on approximately 8.35 million instances from CodeSearchNet and additional GitHub data across eight programming languages. They introduced identifier-aware denoising tasks alongside standard span masking and a bimodal dual generation task using code-comment pairs, then fine-tuned the model on CodeXGLUE benchmark tasks with both task-specific and multi-task learning.

CodeT5 achieved state-of-the-art results on fourteen sub-tasks, including code summarization across six languages, code generation, translation, refinement, defect detection, and clone detection. The base model improved overall summarization scores by more than 1.2 BLEU points over the prior best encoder-decoder model and delivered gains of roughly 4.7 CodeBLEU points on code generation. Identifier-aware pre-training proved especially effective for semantic understanding, while bimodal training particularly benefited natural language to code and code to natural language tasks.

These results indicate that explicitly modeling identifier information and bidirectional natural language-code alignment can produce more capable models for a wide range of software engineering applications, potentially increasing developer productivity and code quality.

The authors recommend adopting the released CodeT5 models and code for downstream applications, while practitioners should treat generated code as drafts requiring expert review. Further work could explore larger-scale training and mitigation of data biases.

Main limitations include the computational resources required for pre-training, possible encoding of social biases from training data, and risks of over-reliance on model outputs or generation of insecure code. Results are supported by extensive experiments across multiple tasks and languages, providing reasonable confidence for the reported performance gains.

Nanyang Technological UniversitySalesforceclone detectionencoder-decodernatural languagesCodeT5Identifier‑Aware Pre‑trainingEncoder‑Decoder Model
arXiv: 2109.00859salesforce/CodeT5
Cover for CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation

Abstract

Pre-trained models for Natural Languages (NL) like BERT and GPT have been recently shown to transfer well to Programming Languages (PL) and largely benefit a broad set of code-related tasks. Despite their success, most current methods either rely on an encoder-only (or decoder-only) pre-training that is suboptimal for generation (resp. understanding) tasks or process the code snippet in the same way as NL, neglecting the special characteristics of PL such as token types. We present CodeT5, a unified pre-trained encoder-decoder Transformer model that better leverages the code semantics conveyed from the developer-assigned identifiers. Our model employs a unified framework to seamlessly support both code understanding and generation tasks and allows for multi-task learning. Besides, we propose a novel identifier-aware pre-training task that enables the model to distinguish which code tokens are identifiers and to recover them when they are masked. Furthermore, we propose to exploit the user-written code comments with a bimodal dual generation task for better NL-PL alignment. Comprehensive experiments show that CodeT5 significantly outperforms prior methods on understanding tasks such as code defect detection and clone detection, and generation tasks across various directions including PL-NL, NL-PL, and PL-PL. Further analysis reveals that our model can better capture semantic information from code. Our code and pre-trained models are released at https: //github.com/salesforce/CodeT5 .

Topics

clone detectionencoder-decodernatural languagesprogramming languagessemantic informationmultitask learningdecodersencoders

Keywords

CodeT5Identifier‑Aware Pre‑trainingEncoder‑Decoder ModelBimodal Dual GenerationCodeXGLUEMasked Span PredictionMasked Identifier PredictionMulti‑Task LearningCode UnderstandingCode Generation

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 CodeT5
  • 3.1 Encoding NL and PL
  • 3.2 Pre-training Tasks
  • 3.3 Fine-tuning CodeT5
  • 4 Experimental Setup
  • 4.1 Pre-training Dataset
  • 4.2 Code-specific Tokenizer
  • 4.3 Downstream Tasks and Metrics
  • 4.4 Comparison Models
  • 4.5 Model Configurations
  • 5 Results and Analysis
  • 5.1 CodeXGLUE Downstream Tasks
  • 5.2 Effects of Bimodal Dual Generation and Multi-task Learning
  • 5.3 Analyzing Identifier-aware Pre-training
  • 6 Conclusion
  • References

Knowls

  1. Knowl 1 — CodeT5 Model Architecture and Pre-training Objectives

    model/method

    CodeT5 is a pre-trained encoder-decoder model built upon the text-to-text transfer Transformer (T5) architecture that unifies programming language (PL) understanding and generation tasks. Unlike standard natural language pre-training, CodeT5 leverages the developer-assigned identifiers extracted from Abstract Syntax Trees (ASTs) as semantic signals.

    CodeT5 accepts unimodal source code sequences x=([CLS],c1,…,cm,[SEP])\mathbf{x} = ([\text{CLS}], c_1, \dots, c_m, [\text{SEP}]) or bimodal natural language and code sequences x=([CLS],w1,…,wn,[SEP],c1,…,cm,[SEP])\mathbf{x} = ([\text{CLS}], w_1, \dots, w_n, [\text{SEP}], c_1, \dots, c_m, [\text{SEP}]). The model is pre-trained using four objectives across unimodal and bimodal data:

    1. Masked Span Prediction (MSP): Randomly masks spans of subwords with sentinel tokens and autoregressively predicts the masked spans at the decoder.
    2. Identifier Tagging (IT): An encoder-based sequence labeling task that predicts whether each code token is an identifier.
    3. Masked Identifier Prediction (MIP): An identifier deobfuscation task that masks all occurrences of unique identifiers with unique sentinel tokens and generates the original identifiers.
    4. Bimodal Dual Generation: A sequence-to-sequence objective that alternates between generating code from natural language descriptions (NL→PL\text{NL} \rightarrow \text{PL}) and generating natural language docstrings from code (PL→NL\text{PL} \rightarrow \text{NL}).

  2. Knowl 2 — Identifier Tagging Objective

    equation

    Identifier Tagging (IT) is an auxiliary encoder-only pre-training objective designed to help CodeT5 capture syntactic token types and data flow structures. For a code sequence of mm tokens (c1,…,cm)(c_1, \dots, c_m), binary ground-truth labels y∈{0,1}m\mathbf{y} \in \{0, 1\}^m are constructed via Abstract Syntax Tree (AST) parsing, where yi=1y_i = 1 if code token cic_i is an identifier (e.g., function name or variable) and yi=0y_i = 0 otherwise (with reserved keywords excluded).

    The final hidden states of the code segment from the encoder parameterized by θe\theta_e are projected to predicted probabilities p=(p1,…,pm)\mathbf{p} = (p_1, \dots, p_m), where pi∈[0,1]p_i \in [0, 1] is the predicted probability that token cic_i is an identifier. The sequence labeling loss is defined by the binary cross-entropy:

    LIT(θe)=∑i=1m−[yilog⁡pi+(1−yi)log⁡(1−pi)]\mathcal{L}_{\text{IT}}(\theta_e) = \sum_{i=1}^m -\left[ y_i \log p_i + (1 - y_i) \log(1 - p_i) \right]

  3. Knowl 3 — Masked Identifier Prediction Objective

    equation

    Masked Identifier Prediction (MIP) is a sequence-to-sequence deobfuscation pre-training task. In contrast to standard span corruption, all occurrences of each unique identifier in the programming language segment are replaced with a single distinct sentinel token (creating a many-to-one mapping from token occurrences to sentinel tags). The target sequence I=(I1,I2,…,I∣I∣)I = (I_1, I_2, \dots, I_{|I|}) is constructed by arranging each unique identifier paired with its assigned sentinel token.

    Given the obfuscated input sequence x\I\mathbf{x}^{\backslash I} and model parameters θ\theta, the decoder is trained to autoregressively reconstruct the sequence of unique identifiers according to the negative log-likelihood loss:

    LMIP(θ)=∑j=1∣I∣−log⁡Pθ(Ij∣x\I,I<j)\mathcal{L}_{\text{MIP}}(\theta) = \sum_{j=1}^{|I|} - \log P_\theta(I_j \mid \mathbf{x}^{\backslash I}, I_{<j})

    where I<jI_{<j} denotes the sequence of target identifier tokens generated before step jj.

  4. Knowl 4 — Bimodal Dual Generation Objective

    model/method

    Bimodal Dual Generation is a pre-training objective designed to align natural language (NL) descriptions and programming language (PL) code. While span prediction exposes the decoder only to discrete sentinel tokens and masked spans, bimodal dual generation trains the model for full-sequence bidirectional translation.

    For each parallel (NL,PL)(\text{NL}, \text{PL}) instance in bimodal corpora (such as functions paired with docstrings), two training samples are formed:

    1. NL→PL\text{NL} \rightarrow \text{PL} generation: given the NL input prefixed with a target language identifier tag (e.g., <java>), the decoder generates the complete source code snippet.
    2. PL→NL\text{PL} \rightarrow \text{NL} generation: given the PL input prefixed with a target language identifier tag (e.g., <en>), the decoder generates the complete natural language text.

    Both generation directions are optimized simultaneously using standard autoregressive cross-entropy loss.

  5. Knowl 5 — Multi-Task Balanced Sampling for CodeT5 Fine-Tuning

    equation

    During multi-task fine-tuning of CodeT5 across NN distinct downstream tasks, task control code prompts (e.g., "Translate Java to CSharp:") are prepended to the source input sequence. To prevent dataset size disparities from biasing the shared model parameters towards high-resource tasks, instances are drawn from a multinomial sampling distribution {qi}i=1N\{q_i\}_{i=1}^N defined by:

    qi=riα∑j=1Nrjα,where ri=ni∑k=1Nnkq_i = \frac{r_i^\alpha}{\sum_{j=1}^N r_j^\alpha}, \quad \text{where } r_i = \frac{n_i}{\sum_{k=1}^N n_k}

    Here nin_i represents the number of training examples available for the ii-th task, rir_i is the relative dataset fraction, and the temperature hyperparameter α\alpha is set to 0.70.7.

  6. Knowl 6 — Code-Specific Byte-Level BPE Tokenizer

    model/method

    CodeT5 implements a custom Byte-level Byte-Pair Encoding (BPE) tokenizer with a vocabulary size of 32,000, trained directly on unimodal and bimodal code datasets after filtering low-frequency tokens (occurring fewer than 3 times) and non-printable characters.

    The tokenizer includes dedicated special tokens: [PAD], [CLS], [SEP], and 100 sentinel tokens [MASK0] through [MASK99].

    Compared to the default natural language T5 tokenizer—which encodes syntax-critical symbols such as curly brackets ({, }) into unknown tokens—the code-specific tokenizer retains programming symbols and reduces tokenized sequence lengths on downstream code benchmarks by 30% to 45%, accelerating training and inference.

  7. Knowl 7 — CodeT5 Performance on Code Summarization and Generation

    data/table

    CodeT5 was evaluated on CodeXGLUE generation tasks against encoder-only models (RoBERTa, CodeBERT, GraphCodeBERT, DOBF), decoder-only models (GPT-2, CodeGPT-2, CodeGPT-adapted), and the BART-based encoder-decoder model PLBART. Code summarization is measured by smoothed BLEU-4 across six programming languages (Ruby, JavaScript, Go, Python, Java, PHP). Code generation on the Concode Java benchmark is measured by Exact Match (EM), BLEU-4, and CodeBLEU.

    Model Ruby JS Go Python Java PHP Overall BLEU
    RoBERTa 11.17 11.90 17.72 18.14 16.47 24.02 16.57
    CodeBERT 12.16 14.90 18.07 19.06 17.65 25.16 17.83
    PLBART 14.11 15.56 18.91 19.30 18.45 23.58 18.32
    CodeT5-small (60M) 14.87 15.32 19.25 20.04 19.92 25.46 19.14
    CodeT5-small + dual-gen 15.30 15.61 19.74 19.94 19.78 26.48 19.48
    CodeT5-base (220M) 15.24 16.16 19.56 20.01 20.31 26.03 19.55
    CodeT5-base + dual-gen 15.73 16.00 19.71 20.11 20.41 26.53 19.75
    Model (Concode) EM BLEU CodeBLEU
    GPT-2 17.35 25.37 29.69
    CodeGPT-2 18.25 28.69 32.71
    CodeGPT-adapted 20.10 32.79 35.98
    PLBART 18.75 36.69 38.52
    CodeT5-small 21.55 38.13 41.39
    CodeT5-small + dual-gen 19.95 39.02 42.21
    CodeT5-base 22.30 40.73 43.20
    CodeT5-base + dual-gen 22.70 41.48 44.10

    CodeT5-small outperforms previous models despite having fewer parameters (60M vs. PLBART's 140M), and CodeT5-base establishes new state-of-the-art results across all generation metrics.

  8. Knowl 8 — CodeT5 Performance on Code Translation, Refinement, and Understanding Tasks

    data/table

    Performance of CodeT5 on code-to-code translation (Java ↔\leftrightarrow C#), code refinement (bug fixing on small and medium Java functions), code defect detection (C language accuracy), and clone detection (Java BigCloneBench F1 score).

    Model Java to C# C# to Java Refine Small Refine Medium
    BLEU EM BLEU EM BLEU EM BLEU EM
    Naive Copy 18.54 0.00 18.69 0.00 78.06 0.00 90.91 0.00
    CodeBERT 79.92 59.00 72.14 58.80 77.42 16.40 91.07 5.20
    GraphCodeBERT 80.58 59.40 72.64 58.80 80.02 17.30 91.31 9.10
    PLBART 83.02 64.60 78.35 65.00 77.02 19.21 88.50 8.98
    CodeT5-small 82.98 64.10 79.10 65.60 76.23 19.06 89.20 10.92
    CodeT5-base 84.03 65.90 79.87 66.90 77.43 21.61 87.64 13.96
    Model Defect Detection (Acc) Clone Detection (F1)
    RoBERTa 61.05 94.9
    CodeBERT 62.08 96.5
    GraphCodeBERT - 97.1
    PLBART 63.18 97.2
    CodeT5-small 63.40 97.1
    CodeT5-base 65.78 97.2

    CodeT5 achieves strong gains on semantic bug-fixing (refinement medium exact match increases from 9.10% with GraphCodeBERT and 8.98% with PLBART to 13.96% with CodeT5-base) and achieves 65.78% accuracy on defect detection while remaining competitive on clone detection.

  9. Knowl 9 — Ablation of Identifier-Aware Pre-training Tasks

    empirical result

    An ablation study on CodeT5-small evaluates the distinct functional contributions of Masked Span Prediction (MSP), Identifier Tagging (IT), and Masked Identifier Prediction (MIP) across Python Code Summarization (BLEU), Concode Code Generation (CodeBLEU), Code Refinement Small (Exact Match), and Defect Detection (Accuracy):

    Variant Sum-PY (BLEU) Code-Gen (CodeBLEU) Refine Small (EM) Defect (Acc)
    Full CodeT5 20.04 41.39 19.06 63.40
    −- MSP 18.93 37.44 15.92 64.02
    −- IT 19.73 39.21 18.65 63.29
    −- MIP 19.81 38.25 18.32 62.92

    Key observations:

    1. Removing MSP causes the largest drops on generation tasks (Code-Gen CodeBLEU drops by 3.95 points and Refinement EM by 3.14 points), indicating MSP is vital for syntactic modeling.
    2. Removing MIP causes the largest performance drop on defect detection (accuracy drops from 63.40% to 62.92%), showing that identifier deobfuscation drives semantic code understanding.
    3. Identifier Tagging (IT) independently achieves >99% F1 across all programming languages, validating that CodeT5 reliably distinguishes identifier tokens from operators and keywords.

  10. Knowl 10 — Mapping Dynamics and Interference between MSP and MIP

    empirical result

    Masked Span Prediction (MSP) and Masked Identifier Prediction (MIP) share the same pool of sentinel tokens but enforce different mapping topologies: MSP employs a 1-to-1 mapping (each sentinel token represents a distinct corrupted span), whereas MIP employs a many-to-one mapping (all occurrences of a specific variable/function identifier share the exact same sentinel token).

    Evaluating models pre-trained on a Java subset of CodeSearchNet reveals significant cross-task asymmetry:

    • MSP-only pre-training: Yields 50.13% accuracy and a 99.80% sentinel number matching rate on MSP tasks, but collapses to 2.94% accuracy and a 1.60% matching rate when tested on MIP tasks.
    • MIP-only pre-training: Yields 42.75% accuracy and a 98.80% match rate on MIP tasks, while maintaining an 82.40% sentinel match rate on MSP tasks (though achieving only 1.68% accuracy due to span length mismatch).
    • Joint MIP + MSP pre-training: Achieves 48.26% accuracy on MSP (99.60% matching rate) and 42.72% accuracy on MIP (98.60% matching rate), reconciling the 1-to-1 and many-to-one mapping dynamics without destructive interference.

Coverage note — None was omitted; all contributed models, pre-training objectives, tokenization designs, multi-task formulations, benchmark results on CodeXGLUE, and ablation analyses are covered.

References

  1. 1.Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified pre-training for program understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 2655–2668. Association for Computational Linguistics.
  2. 2.Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code. CoRR, abs/2107.03374.
  3. 3.Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pre-training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
  4. 4.Colin B. Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. 2020. Pymt5: multi-mode translation of natural language and python code with transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 9052–9065. Association for Computational Linguistics.
  5. 5.Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 7057–7067.
  6. 6.Sergio Cozzetti B. de Souza, Nicolas Anquetil, and Káthia Marçal de Oliveira. 2005. A study of the documentation essential to software maintenance. In Proceedings of the 23rd Annual International Conference on Design of Communication: documenting & Designing for Pervasive Information, SIGDOC 2005, Coventry, UK, September 21-23, 2005, pages 68–75. ACM.
  7. 7.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186.
  8. 8.Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13042–13054.
  9. 9.Ahmed Elnaggar, Wei Ding, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Silvia Severini, Florian Matthes, and Burkhard Rost. 2021. Codetrans: Towards cracking the language of silicone’s code through self-supervised deep learning and high performance computing. CoRR, abs/2104.02443.
  10. 10.Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Codebert: A pre-trained model for programming and natural languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16-20 November 2020, pages 1536–1547. Association for Computational Linguistics.
  11. 11.Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin B. Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. Graphcodebert: Pre-training code representations with data flow. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
  12. 12.Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Codesearchnet challenge: Evaluating the state of semantic code search. CoRR, abs/1909.09436.
  13. 13.Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 1643–1652. Association for Computational Linguistics.
  14. 14.Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5110–5121. PMLR.
  15. 15.Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7871–7880. Association for Computational Linguistics.
  16. 16.Chin-Yew Lin and Franz Josef Och. 2004. ORANGE: a method for evaluating automatic evaluation metrics for machine translation. In COLING 2004, 20th International Conference on Computational Linguistics, Proceedings of the Conference, 23-27 August 2004, Geneva, Switzerland.
  17. 17.Fang Liu, Ge Li, Yunfei Zhao, and Zhi Jin. 2020. Multi-task learning based pre-trained language model for code completion. In 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020, Melbourne, Australia, September 21-25, 2020, pages 473–485. IEEE.
  18. 18.Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4487–4496. Association for Computational Linguistics.
  19. 19.Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
  20. 20.Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. CoRR, abs/2102.04664.
  21. 21.Antonio Mastropaolo, Simone Scalabrino, Nathan Cooper, David Nader-Palacio, Denys Poshyvanyk, Rocco Oliveto, and Gabriele Bavota. 2021. Studying the usage of text-to-text transfer transformer to support code-related tasks. In 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021, pages 336–347. IEEE.
  22. 22.Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  23. 23.Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67.
  24. 24.Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. CoRR, abs/2009.10297.
  25. 25.Baptiste Rozière, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  26. 26.Baptiste Rozière, Marie-Anne Lachaux, Marc Szafraniec, and Guillaume Lample. 2021. DOBF: A deobfuscation pre-training objective for programming languages. CoRR, abs/2102.07492.
  27. 27.Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
  28. 28.Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. MASS: masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5926–5936. PMLR.
  29. 29.Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. ERNIE: enhanced representation through knowledge integration. CoRR, abs/1904.09223.
  30. 30.Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020. Intellicode compose: code generation using transformer. In ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, pages 1433–1443. ACM.
  31. 31.Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. 2019. An empirical study on learning bug-fixing patches in the wild via neural machine translation. ACM Trans. Softw. Eng. Methodol., 28(4):19:1–19:29.
  32. 32.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008.
  33. 33.Wenhan Wang, Ge Li, Bo Ma, Xin Xia, and Zhi Jin. 2020. Detecting code clones with graph neural network and flow-augmented abstract syntax tree. In 27th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2020, London, ON, Canada, February 18-21, 2020, pages 261–271. IEEE.
  34. 34.Yaqin Zhou, Shangqing Liu, Jing Kai Siow, Xiaoning Du, and Yang Liu. 2019. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 10197–10207.
  35. 35.Daniel Zügner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan Günnemann. 2021. Language-agnostic representation learning of source code from structure and context. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.

Citation

MLA
Wang, Y., et al. “CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation”. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 8696–708, https://doi.org/10.18653/v1/2021.emnlp-main.685.
APA
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021). CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 8696–8708. https://doi.org/10.18653/v1/2021.emnlp-main.685
Chicago
Wang, Y., W. Wang, S. Joty, and S. C. H. Hoi. 2021. “CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation”. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 8696–8708. https://doi.org/10.18653/v1/2021.emnlp-main.685.
Harvard
Wang, Y. et al. (2021) “CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation”, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp. 8696–8708. Available at: https://doi.org/10.18653/v1/2021.emnlp-main.685.
Vancouver
1. Wang Y, Wang W, Joty S, Hoi SCH (2021) CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp 8696–8708

BibTeX

@inproceedings{wang-etal-2021-codet5,
    title = "{C}ode{T}5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation",
    author = "Wang, Yue  and
      Wang, Weishi  and
      Joty, Shafiq  and
      Hoi, Steven C.H.",
    editor = "Moens, Marie-Francine  and
      Huang, Xuanjing  and
      Specia, Lucia  and
      Yih, Scott Wen-tau",
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.685/",
    doi = "10.18653/v1/2021.emnlp-main.685",
    pages = "8696--8708"
}
Metadata:ACL Anthology

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/