Back

GraphCodeBERT: Pre-training Code Representations with Data Flow

Daya GuoShuo RenShuai LuZhangyin FengDuyu TangShujie LiuLong ZhouNan DuanJian YinDaxin Jiang

article2020ICLR1,882 citations

Introduces GraphCodeBERT, a pre-trained Transformer model that incorporates variable data-flow graphs to capture semantic code structure, achieving state-of-the-art performance across code search, clone detection, and translation without the overhead of deep abstract syntax trees.

Listen

Modern software engineering increasingly relies on artificial intelligence to assist with developer workflows, such as searching repositories, detecting duplicate code, translating between languages, and fixing bugs. However, most existing pre-trained models treat programming code merely as flat sequences of text, ignoring the critical structural dependencies between variables. While some alternative methods attempt to incorporate syntactic structures like Abstract Syntax Trees, those hierarchies are often too complex and computationally expensive. The article demonstrates a novel pre-trained model, GraphCodeBERT, which incorporates lightweight semantic data flows—tracking where variable values originate—to significantly improve automated code understanding and generation.

To evaluate this approach, the researchers pre-trained GraphCodeBERT on a dataset of 2.3 million function-document pairs across six programming languages (Python, Java, JavaScript, PHP, Ruby, and Go). The model uses a standard Transformer architecture augmented with a graph-guided masked attention mechanism. In addition to standard masked language modeling, the training incorporated two novel structure-aware tasks: predicting masked data flow connections between variables and aligning variable nodes in the graph with their corresponding source code tokens. The model was then benchmarked across four downstream applications: natural language code search, clone detection, cross-language code translation, and automated code refinement.

GraphCodeBERT established new state-of-the-art performance across all four evaluated tasks. In natural language code search, the model achieved an overall Mean Reciprocal Rank of 0.713, outperforming existing strong baselines like CodeBERT (0.693) with high statistical significance. In code clone detection, it matched top-tier specialized models with an F1 score of 0.950 while exceeding standard text-only pre-trained models. For code refinement, it improved bug-fixing accuracy on a medium-sized Java benchmark from 5.2% (CodeBERT) to 9.1%—a relative gain of 75%. Ablation studies confirmed that removing data flow representation or the structure-aware pre-training objectives caused measurable drops in performance, and internal attention analyses revealed that the model actively prioritizes variable dependency nodes over raw code tokens.

These findings indicate that incorporating semantic data flow provides a highly efficient and effective way to teach machine learning models how code behaves rather than just how it reads. For organizations building or deploying developer tooling, this approach offers reduced software maintenance costs, better bug-patching accuracy, and faster migration of legacy systems across programming languages, all while avoiding the computational overhead associated with deep syntax trees.

Technical leaders and software tooling teams should consider integrating data-flow-aware representations into code intelligence pipelines. For future development, the article suggests incorporating third-party library definitions and specialized grammar-aware decoders to eliminate remaining errors, such as hallucinations of undefined variables or missing closing syntax during code generation. The results are highly credible within the scope of the six evaluated languages and benchmarks, though organizations should evaluate domain-specific library performance in pilot testing before deploying generative models directly into production codebases.

Beihang UniversityHarbin Institute of TechnologyMicrosoftPeking UniversitySun Yat-sen Universityabstract syntaxclone detectionencoder-decoderPre-training code representationsdata flowabstract syntax tree
arXiv: 2009.08366microsoft/CodeBERT
Cover for GraphCodeBERT: Pre-training Code Representations with Data Flow

Abstract

Pre-trained models for programming language have achieved dramatic empirical improvements on a variety of code-related tasks such as code search, code completion, code summarization, etc. However, existing pre-trained models regard a code snippet as a sequence of tokens, while ignoring the inherent structure of code, which provides crucial code semantics and would enhance the code understanding process. We present GraphCodeBERT, a pre-trained model for programming language that considers the inherent structure of code. Instead of taking syntactic-level structure of code like abstract syntax tree (AST), we use data flow in the pre-training stage, which is a semantic-level structure of code that encodes the relation of "where-the-value-comes-from" between variables. Such a semantic-level structure is neat and does not bring an unnecessarily deep hierarchy of AST, the property of which makes the model more efficient. We develop GraphCodeBERT based on Transformer. In addition to using the task of masked language modeling, we introduce two structure-aware pre-training tasks. One is to predict code structure edges, and the other is to align representations between source code and code structure. We implement the model in an efficient way with a graph-guided masked attention function to incorporate the code structure. We evaluate our model on four tasks, including code search, clone detection, code translation, and code refinement. Results show that code structure and newly introduced pre-training tasks can improve GraphCodeBERT and achieves state-of-the-art performance on the four downstream tasks. We further show that the model prefers structure-level attentions over token-level attentions in the task of code search.

Topics

abstract syntaxclone detectionencoder-decoderlanguage modelmasked language modelingprogramming languagessource codesbinary codescode generationsyntactic structure

Keywords

Pre-training code representationsdata flowabstract syntax treecode searchcode refinementcode summarizationcode translationcode generationcode completionlanguage modeling

Table of Contents

  • 1 Introduction
  • 2 Related Works
  • 3 Data Flow
  • 4 GraphCodeBERT
  • 4.1 Model Architecture
  • 4.2 Graph-guided Masked Attention
  • 4.3 Pre-training Tasks
  • 5 Experiments
  • 5.1 Natural Language Code Search
  • 5.2 Code Clone Detection
  • 5.3 Code Translation
  • 5.4 Code Refinement
  • 5.5 Model Analysis
  • 6 Conclusion
  • References
  • A Pre-training Details
  • B Natural Language Code Search
  • C Code Clone Detection
  • D Code Translation
  • E Code Refinement
  • F Case Study
  • F.1 Natural Language Code Search
  • F.2 Code Clone Detection
  • F.3 Code Translation and Code Refinement
  • G Error Analysis

Knowls

  1. Knowl 1 — GraphCodeBERT Architecture and Input Representation

    model/method

    GraphCodeBERT is a Transformer-based pre-trained model designed to learn representations from both source code and its semantic data flow graph. The input sequence XX is constructed by concatenating natural language comments W={w1,w2,…,wm}W = \{w_1, w_2, \dots, w_m\}, source code tokens C={c1,c2,…,cn}C = \{c_1, c_2, \dots, c_n\}, and data flow variable nodes V={v1,v2,…,vk}V = \{v_1, v_2, \dots, v_k\}:

    X={[CLS],w1,…,wm,[SEP],c1,…,cn,[SEP],v1,…,vk}X = \{[\text{CLS}], w_1, \dots, w_m, [\text{SEP}], c_1, \dots, c_n, [\text{SEP}], v_1, \dots, v_k\}

    where [CLS][\text{CLS}] is a special classification token, and [SEP][\text{SEP}] separates sequence segments.

    For each input token in XX, the input vector H0H^0 is the sum of its token embedding and position embedding. All variable nodes in VV share an identical special position embedding indicating they are nodes of the data flow graph. The model applies NN Transformer layers to produce contextual representations Hn=transformern(Hn−1)H^n = \text{transformer}_n(H^{n-1}) for n∈[1,N]n \in [1, N]:

    Gn=LN(MultiAttn(Hn−1)+Hn−1)G^n = \text{LN}(\text{MultiAttn}(H^{n-1}) + H^{n-1})

    Hn=LN(FFN(Gn)+Gn)H^n = \text{LN}(\text{FFN}(G^n) + G^n)

    where MultiAttn\text{MultiAttn} is multi-headed self-attention, FFN\text{FFN} is a two-layer feed-forward network, and LN\text{LN} denotes layer normalization. For the nn-th layer with hidden dimension dhd_h, head dimension dkd_k, and uu attention heads, the output G^n\hat{G}^n is computed using projection matrices WiQ,WiK,WiV∈Rdh×dkW_i^Q, W_i^K, W_i^V \in \mathbb{R}^{d_h \times d_k} and WnO∈Rdh×dhW_n^O \in \mathbb{R}^{d_h \times d_h} with graph-guided attention mask M∈R∣X∣×∣X∣M \in \mathbb{R}^{|X| \times |X|}:

    Qi=Hn−1WiQ,Ki=Hn−1WiK,Vi=Hn−1WiVQ_i = H^{n-1}W_i^Q, \quad K_i = H^{n-1}W_i^K, \quad V_i = H^{n-1}W_i^V

    headi=softmax(QiKiTdk+M)Vi\text{head}_i = \text{softmax}\left(\frac{Q_i K_i^T}{\sqrt{d_k}} + M\right)V_i

    G^n=[head1;… ;headu]WnO\hat{G}^n = [\text{head}_1; \dots; \text{head}_u]W_n^O

  2. Knowl 2 — Graph-Guided Masked Attention Matrix

    equation

    In GraphCodeBERT, relational structure between tokens and data flow nodes is incorporated into self-attention via a masking matrix M∈R∣X∣×∣X∣M \in \mathbb{R}^{|X| \times |X|}. Let WW be comment tokens, CC be code tokens, and VV be data flow variable nodes in sequence XX. Let EE be the set of directed data flow edges between variables, and let E′E' be the set of alignment edges between variable nodes and code tokens (where ⟨vi,cj⟩∈E′\langle v_i, c_j \rangle \in E' and ⟨cj,vi⟩∈E′\langle c_j, v_i \rangle \in E' if variable node viv_i is identified from code token cjc_j).

    The attention mask entry MijM_{ij} between the ii-th query qiq_i and the jj-th key kjk_j is defined as:

    Mij={0if qi∈{[CLS],[SEP]}∨qi,kj∈W∪C∨⟨qi,kj⟩∈E∪E′∨(qi,kj∈V∧i=j)−∞otherwiseM_{ij} = \begin{cases} 0 & \text{if } q_i \in \{[\text{CLS}], [\text{SEP}]\} \lor q_i, k_j \in W \cup C \lor \langle q_i, k_j \rangle \in E \cup E' \lor (q_i, k_j \in V \land i = j) \\ -\infty & \text{otherwise} \end{cases}

    Under this mask:

    • Special tokens [CLS][\text{CLS}] and [SEP][\text{SEP}] attend to all positions.
    • Natural language comment tokens and code tokens can mutually and freely attend to all tokens within W∪CW \cup C.
    • A variable node qviq_{v_i} can attend to another variable node kvjk_{v_j} only if there is a directed edge from vjv_j to viv_i in EE (i.e. value flows from vjv_j into viv_i) or if i=ji = j.
    • A variable node qviq_{v_i} and code token kcjk_{c_j} can attend to each other if and only if viv_i originates from token cjc_j (⟨vi,cj⟩∈E′\langle v_i, c_j \rangle \in E').
    • All other interactions are set to −∞-\infty, yielding zero attention weight after softmax.

  3. Knowl 3 — Semantic Data Flow Graph Extraction

    model/method

    Data flow is a directed graph G(C)=(V,E)G(C) = (V, E) representing the semantic relation of "where-the-value-comes-from" between variables in source code C={c1,c2,…,cn}C = \{c_1, c_2, \dots, c_n\}. Unlike Abstract Syntax Trees (ASTs), data flow is invariant under different syntactic grammars for equivalent source code and avoids deeply nested hierarchies.

    Data flow is extracted through the following steps:

    1. The source code CC is parsed into an AST using a compiler parser.
    2. The terminal (leaf) nodes of the AST are processed to identify the ordered sequence of variable occurrences, denoted V={v1,v2,…,vk}V = \{v_1, v_2, \dots, v_k\}. Each variable occurrence constitutes a distinct node in the graph.
    3. Directed edges E={ε1,ε2,…,εl}E = \{\varepsilon_1, \varepsilon_2, \dots, \varepsilon_l\} are extracted from assignment and expression syntax. A directed edge ε=⟨vi,vj⟩\varepsilon = \langle v_i, v_j \rangle indicates that the value of variable vjv_j is computed from or originates from variable viv_i (e.g., in x=a+bx = a + b, directed edges ⟨a,x⟩\langle a, x \rangle and ⟨b,x⟩\langle b, x \rangle are added).
    4. The variable-to-source-token alignment edge set E′={⟨vi,cj⟩,⟨cj,vi⟩∣vi is identified from code token cj}E' = \{\langle v_i, c_j \rangle, \langle c_j, v_i \rangle \mid v_i \text{ is identified from code token } c_j\} is maintained to link the data flow graph to textual code tokens.

  4. Knowl 4 — Structure-Aware Pre-Training: Edge Prediction and Node Alignment Objectives

    equation

    GraphCodeBERT introduces two structure-aware pre-training objectives alongside Masked Language Modeling (MLM):

    1. Data Flow Edge Prediction: Encourages the model to learn variable dependency representations. A subset Vs⊂VV_s \subset V consisting of 20% of variable nodes is randomly sampled, and direct edges connected to these nodes are masked in the attention mask matrix. Candidate pairs are Ec=(Vs×V)∪(V×Vs)E_c = (V_s \times V) \cup (V \times V_s). The probability peijp_{e_{ij}} of an edge existing from node viv_i to vjv_j is computed via dot product followed by a sigmoid function σ(hviThvj)\sigma(h_{v_i}^T h_{v_j}), where hvh_v is the contextual representation of variable node vv. With balanced positive and negative sampling:

    lossEdgePred=−∑eij∈Ec[δ(eij∈Emask)log⁡peij+(1−δ(eij∈Emask))log⁡(1−peij)]\text{loss}_{\text{EdgePred}} = - \sum_{e_{ij} \in E_c} \left[ \delta(e_{ij} \in E_{\text{mask}}) \log p_{e_{ij}} + (1 - \delta(e_{ij} \in E_{\text{mask}})) \log (1 - p_{e_{ij}}) \right]

    where δ(eij∈Emask)\delta(e_{ij} \in E_{\text{mask}}) is 11 if directed edge ⟨vi,vj⟩\langle v_i, v_j \rangle belongs to the masked true edge set EmaskE_{\text{mask}} and 00 otherwise.

    1. Node Alignment: Aligns variable nodes in data flow with corresponding code tokens in source code. Edges between a 20% random sample of nodes Vs′⊂VV'_s \subset V and code tokens CC are masked (Emask′E'_{\text{mask}}). For candidate set Ec′=Vs′×CE'_c = V'_s \times C with balanced positive and negative samples, and peij=σ(hviThcj)p_{e_{ij}} = \sigma(h_{v_i}^T h_{c_j}):

    lossNodeAlign=−∑eij∈Ec′[δ(eij∈Emask′)log⁡peij+(1−δ(eij∈Emask′))log⁡(1−peij)]\text{loss}_{\text{NodeAlign}} = - \sum_{e_{ij} \in E'_c} \left[ \delta(e_{ij} \in E'_{\text{mask}}) \log p_{e_{ij}} + (1 - \delta(e_{ij} \in E'_{\text{mask}})) \log (1 - p_{e_{ij}}) \right]

  5. Knowl 5 — CodeSearchNet Natural Language Code Search Results

    data/table

    Models were evaluated on natural language code search across six programming languages from the CodeSearchNet corpus. Relevance scores are calculated as the inner product of [CLS][\text{CLS}] encodings between natural language queries and code candidates, evaluated across the full candidate corpus using Mean Reciprocal Rank (MRR):

    Model Ruby Javascript Go Python Java Php Overall
    NBow 0.162 0.157 0.330 0.161 0.171 0.152 0.189
    CNN 0.276 0.224 0.680 0.242 0.263 0.260 0.324
    BiRNN 0.213 0.193 0.688 0.290 0.304 0.338 0.338
    selfAtt 0.275 0.287 0.723 0.398 0.404 0.426 0.419
    RoBERTa 0.587 0.517 0.850 0.587 0.599 0.560 0.617
    RoBERTa (code) 0.628 0.562 0.859 0.610 0.620 0.579 0.643
    CodeBERT 0.679 0.620 0.882 0.672 0.676 0.628 0.693
    GraphCodeBERT 0.703 0.644 0.897 0.692 0.691 0.649 0.713

    GraphCodeBERT outperforms CodeBERT by 2.0% absolute overall MRR (p<0.01p < 0.01). Under the original CodeSearchNet setting with 1,000 candidate codes per query, GraphCodeBERT achieves an overall MRR of 0.774 (Ruby: 0.732, Javascript: 0.711, Go: 0.841, Python: 0.879, Java: 0.757, Php: 0.725), compared to CodeBERT's overall MRR of 0.760.

  6. Knowl 6 — Code Clone Detection Performance on BigCloneBench

    data/table

    Code clone detection evaluates binary classification of whether two source code fragments perform identical functionality on the BigCloneBench benchmark (901,724 training, 416,328 validation, 416,328 test pairs):

    Model Precision Recall F1
    Deckard 0.93 0.02 0.03
    RtvNN 0.95 0.01 0.01
    CDLH 0.92 0.74 0.82
    ASTNN 0.92 0.94 0.93
    FA-AST-GMN 0.96 0.94 0.95
    RoBERTa (code) 0.949 0.922 0.935
    CodeBERT 0.947 0.934 0.941
    GraphCodeBERT 0.948 0.952 0.950

    GraphCodeBERT significantly outperforms pre-trained sequence models without code structure (CodeBERT at 0.941 F1 and RoBERTa at 0.935 F1, p<0.01p < 0.01), while matching the performance of dedicated graph neural network methods on flow-augmented ASTs (FA-AST-GMN at 0.950 F1) with higher recall (0.952 vs. 0.94).

  7. Knowl 7 — Code Translation and Code Refinement Performance

    data/table

    GraphCodeBERT was evaluated as an encoder in sequence-to-sequence Transformer models for Java-to-C# and C#-to-Java code translation (11,800 method pairs), and code refinement (bug fixing) on small and medium datasets (Tufano et al.):

    Java →\rightarrow C# C# →\rightarrow Java
    Method BLEU Accuracy (%) BLEU Accuracy (%)
    Naive 18.54 0.0 18.69 0.0
    PBSMT 43.53 12.5 40.06 16.1
    Transformer 55.84 33.0 50.47 37.9
    RoBERTa (code) 77.46 56.1 71.99 57.9
    CodeBERT 79.92 59.0 72.14 58.8
    GraphCodeBERT 80.58 59.4 72.64 58.8

    For code refinement on buggy Java functions:

    Small (length ≤50\le 50) Medium (length 50−10050-100)
    Method BLEU Accuracy (%) BLEU Accuracy (%)
    Naive 78.06 0.0 90.91 0.0
    LSTM 76.76 10.0 72.08 2.5
    Transformer 77.21 14.7 89.25 3.7
    RoBERTa (code) 77.30 15.9 90.07 4.1
    CodeBERT 77.42 16.4 91.07 5.2
    GraphCodeBERT 80.02 17.3 91.31 9.1

    GraphCodeBERT achieves the highest translation and refinement scores, increasing exact fix accuracy on medium refinement from 5.2% (CodeBERT) to 9.1%.

  8. Knowl 8 — Ablation of Data Flow and Structure-Aware Pre-Training Objectives

    empirical result

    An ablation study on natural language code search across six languages evaluated the individual contributions of data flow input and the two structure-aware pre-training objectives:

    Methods Ruby Javascript Go Python Java Php Overall
    GraphCodeBERT 0.703 0.644 0.897 0.692 0.691 0.649 0.713
    -w/o EdgePred 0.701 0.632 0.894 0.687 0.688 0.640 0.707
    -w/o NodeAlign 0.685 0.635 0.887 0.682 0.690 0.640 0.703
    -w/o Data Flow 0.679 0.620 0.882 0.672 0.676 0.628 0.693

    Removing the data flow input entirely drops overall MRR from 0.713 to 0.693 (-2.0%). Removing the NodeAlign pre-training task reduces MRR to 0.703 (-1.0%), and removing EdgePred reduces MRR to 0.707 (-0.6%), showing that both edge prediction and code-variable alignment objectives provide complementary improvements to the base data flow representation.

  9. Knowl 9 — Attention Distribution Analysis Between Data Flow Nodes and Code Tokens

    empirical result

    Analysis of self-attention weights from the [CLS][\text{CLS}] classification token across language test sets demonstrates that GraphCodeBERT learns a strong preference for attending to data flow nodes relative to code tokens:

    Distribution (%) Ruby Javascript Go Python Java Php
    Codes / Nodes ratio 90.1 / 9.9 94.6 / 5.4 95.0 / 5.03 80.6 / 19.4 93.2 / 6.8 87.5 / 12.5
    [CLS]→[\text{CLS}] \rightarrow Codes / Nodes 82.3 / 17.7 89.7 / 10.3 91.0 / 9.0 67.7 / 32.3 87.8 / 12.2 79.4 / 20.6

    While data flow nodes account for only 5.0% to 19.4% of token positions in the input sequence, [CLS][\text{CLS}] directs 9.0% to 32.3% of its total attention weight to these nodes, indicating that semantic data flow edges provide concentrated signals for overall code understanding.

  10. Knowl 10 — Comparison of Data Flow and AST Structure Representations

    empirical result

    Evaluating retrieval MRR on Ruby code search with varying sequence lengths (64 to 512 tokens) reveals key differences between data flow and Abstract Syntax Tree (AST) representations:

    1. AST Pre-order Traversal (linearizing all AST nodes) and AST Subtree Masking (restricting node-query self-attention to subtree descendants) degrade MRR at short input lengths (≤128\le 128) compared to the baseline without code structure, only matching or slightly exceeding it at length 512.
    2. GraphCodeBERT (Data Flow) consistently outperforms both the structure-free baseline and AST-based variants across all input sequence lengths, maintaining an MRR advantage from length 64 through length 512 (achieving >0.74>0.74 MRR at 512 tokens).
    3. Data flow is more efficient and accurate because variable nodes represent only 5% to 20% of total tokens and encode direct value flows, whereas AST introduces an extensive hierarchy of syntactic nodes that consumes sequence capacity and incurs O(n2)O(n^2) self-attention overhead.

Coverage note — None was omitted; the extracted knowls comprehensively capture the data flow definition and extraction, model architecture, graph-guided attention masking, pre-training objectives, pre-training configuration, all four downstream experimental benchmarks, ablation studies, attention allocation analyses, and AST comparisons.

References

  1. 1.Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. Learning to represent programs with graphs. In International Conference on Learning Representations, 2018.
  2. 2.Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. code2seq: Generating sequences from structured representations of code. arXiv preprint arXiv:1808.01400, 2018.
  3. 3.Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. arXiv, pp. arXiv–1910, 2019.
  4. 4.Marc Brockschmidt, Miltiadis Allamanis, Alexander L Gaunt, and Oleksandr Polozov. Generative code modeling with graphs. arXiv preprint arXiv:1805.08490, 2018.
  5. 5.Luca Buratti, Saurabh Pujar, Mihaela Bornea, Scott McCarley, Yunhui Zheng, Gaetano Rossiello, Alessandro Morari, Jim Laredo, Veronika Thost, Yufan Zhuang, et al. Exploring software naturalness throughneural language models. arXiv preprint arXiv:2006.12641, 2020.
  6. 6.Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree neural networks for program translation. In Advances in neural information processing systems, pp. 2547–2557, 2018.
  7. 7.Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab S Mirrokni. Locality-sensitive hashing scheme based on p-stable distributions. In Proceedings of the twentieth annual symposium on Computational geometry, pp. 253–262, 2004.
  8. 8.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  9. 9.Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155, 2020.
  10. 10.Daya Guo, Duyu Tang, Nan Duan, M. Zhou, and Jian Yin. Dialog-to-action: Conversational question answering over a large-scale knowledge base. In NeurIPS, 2018.
  11. 11.Daya Guo, Duyu Tang, Nan Duan, M. Zhou, and Jian Yin. Coupling retrieval and meta-learning for context-dependent semantic parsing. ArXiv, abs/1906.07108, 2019.
  12. 12.Vincent J Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. Global relational models of source code. In International Conference on Learning Representations, 2019.
  13. 13.Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. Deep code comment generation. In 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC), pp. 200–20010. IEEE, 2018.
  14. 14.Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436, 2019.
  15. 15.Lingxiao Jiang, Ghassan Misherghi, Zhendong Su, and Stephane Glondu. Deckard: Scalable and accurate tree-based detection of code clones. In 29th International Conference on Software Engineering (ICSE’07), pp. 96–105. IEEE, 2007.
  16. 16.Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Pre-trained contextual embedding of source code. arXiv preprint arXiv:2001.00059, 2019.
  17. 17.Svetoslav Karaivanov, Veselin Raychev, and Martin Vechev. Phrase-based statistical translation of programming languages. In Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software, pp. 173–184, 2014.
  18. 18.Rafael-Michael Karampatsis and Charles Sutton. Scelmo: Source code embeddings from language models. arXiv preprint arXiv:2004.13214, 2020.
  19. 19.Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. Code prediction by feeding trees to transformers. arXiv preprint arXiv:2003.13848, 2020.
  20. 20.Philipp Koehn, Franz J Och, and Daniel Marcu. Statistical phrase-based translation. Technical report, UNIVERSITY OF SOUTHERN CALIFORNIA MARINA DEL REY INFORMATION SCIENCES INST, 2003.
  21. 21.Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019.
  22. 22.Jian Li, Yue Wang, Michael R Lyu, and Irwin King. Code completion with neural attention and pointer networks. arXiv preprint arXiv:1711.09573, 2017.
  23. 23.Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  24. 24.Anh Tuan Nguyen and Tien N Nguyen. Graph-based statistical language model for code. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, volume 1, pp. 858–868. IEEE, 2015.
  25. 25.Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. Lexical statistical machine translation for language migration. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, pp. 651–654, 2013.
  26. 26.Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. Divide-and-conquer approach for multi-phase statistical migration for source code (t). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 585–596. IEEE, 2015.
  27. 27.Xuan-Phi Nguyen, Shafiq Joty, Steven Hoi, and Richard Socher. Tree-structured attention with hierarchical accumulation. In International Conference on Learning Representations, 2019.
  28. 28.Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018.
  29. 29.Maxim Rabinovich, Mitchell Stern, and Dan Klein. Abstract syntax networks for code generation and semantic parsing. arXiv preprint arXiv:1704.07535, 2017.
  30. 30.Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openaiassets/researchcovers/languageunsupervised/language understanding paper. pdf, 2018.
  31. 31.Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.
  32. 32.Jeffrey Svajlenko, Judith F Islam, Iman Keivanloo, Chanchal K Roy, and Mohammad Mamun Mia. Towards a big data curated benchmark of inter-project code clones. In 2014 IEEE International Conference on Software Maintenance and Evolution, pp. 476–480. IEEE, 2014.
  33. 33.Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. Intellicode compose: Code generation using transformer. arXiv preprint arXiv:2005.08025, 2020.
  34. 34.Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. An empirical study on learning bug-fixing patches in the wild via neural machine translation. ACM Transactions on Software Engineering and Methodology (TOSEM), 28(4):1–29, 2019.
  35. 35.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008, 2017.
  36. 36.Wenhan Wang, Ge Li, Bo Ma, Xin Xia, and Zhi Jin. Detecting code clones with graph neural networkand flow-augmented abstract syntax tree. arXiv preprint arXiv:2002.08653, 2020.
  37. 37.Huihui Wei and Ming Li. Supervised deep features for software functional clone detection by exploiting lexical and syntactical information in source code. In IJCAI, pp. 3034–3040, 2017.
  38. 38.Martin White, Michele Tufano, Christopher Vendome, and Denys Poshyvanyk. Deep learning code fragments for code clone detection. In 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 87–98. IEEE, 2016.
  39. 39.Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237, 2019.
  40. 40.Pengcheng Yin and Graham Neubig. A syntactic neural model for general-purpose code generation. In The 55th Annual Meeting of the Association for Computational Linguistics (ACL), Vancouver, Canada, July 2017. URL https://arxiv.org/abs/1704.01696.
  41. 41.Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. A novel neural source code representation based on abstract syntax tree. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 783–794. IEEE, 2019.

Citation

MLA
Guo, D., et al. “GraphCodeBERT: Pre-training Code Representations with Data Flow”. arXiv, 2020, http://arxiv.org/abs/2009.08366v4.
APA
Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., Tufano, M., Deng, S. K., Clement, C., Drain, D., Sundaresan, N., Yin, J., Jiang, D., & Zhou, M. (2020). GraphCodeBERT: Pre-training Code Representations with Data Flow. arXiv. http://arxiv.org/abs/2009.08366v4
Chicago
Guo, D., S. Ren, S. Lu, et al. 2020. “GraphCodeBERT: Pre-training Code Representations with Data Flow”. arXiv. http://arxiv.org/abs/2009.08366v4.
Harvard
Guo, D. et al. (2020) “GraphCodeBERT: Pre-training Code Representations with Data Flow”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2009.08366v4.
Vancouver
1. Guo D, Ren S, Lu S, et al (2020) GraphCodeBERT: Pre-training Code Representations with Data Flow. arXiv

BibTeX

@article{guo2020graphcodebert,
  title = {GraphCodeBERT: Pre-training Code Representations with Data Flow},
  author = {Guo, Daya and Ren, Shuo and Lu, Shuai and Feng, Zhangyin and Tang, Duyu and Liu, Shujie and Zhou, Long and Duan, Nan and Svyatkovskiy, Alexey and Fu, Shengyu and Tufano, Michele and Deng, Shao Kun and Clement, Colin and Drain, Dawn and Sundaresan, Neel and Yin, Jian and Jiang, Daxin and Zhou, Ming},
  year = {2020},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2009.08366v4},
  eprint = {2009.08366}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Published with permission