Back

Program Synthesis with Large Language Models

Jacob AustinAugustus OdenaMaxwell NyeMaarten BosmaHenryk MichalewskiDavid DohanEllen JiangCarrie CaiMichael TerryQuoc Le

article2021arXiv4,460 citations

Introduces the MBPP benchmark for Python code synthesis, demonstrating that large language model performance scales log-linearly with model size and that conversational human feedback cuts error rates in half.

Listen

The growing capability of large language models offers promising avenues for automating software engineering workflows. Historically, automated program synthesis has been largely confined to narrow, domain-specific languages due to the difficulty of producing correct, executable code in general-purpose languages. Developing models that can generate functional Python code directly from human descriptions has major real-world significance, as it could substantially boost developer productivity and make software development more accessible to non-experts.

The article evaluates the extent to which dense, decoder-only Transformer language models can synthesize short, functional Python programs from natural language prompts and input-output assertions. It also examines the models' capacity to refine code interactively with human feedback and their ability to execute and predict the runtime output of existing programs.

The evaluation used language models ranging from 244 million to 137 billion parameters, pre-trained on a broad web and dialog corpus without specialized code pre-training. Researchers created two primary benchmarks: Mostly Basic Programming Problems (MBPP), comprising 974 entry-level Python programming tasks with functional unit tests, and MathQA-Python, containing 23,914 mathematical word problems translated into Python. The evaluation assessed both few-shot prompting and fine-tuned models, testing functional correctness by executing the generated solutions against rigorous test suites using multiple sampling temperatures.

Synthesis performance scales predictably and log-linearly with model size across both datasets. On MBPP, the largest 137B model solved 59.6% of tasks using few-shot prompting, and fine-tuning improved accuracy by approximately 10 percentage points across most model scales. On MathQA-Python, the fine-tuned 137B model attained 83.8% accuracy. Crucially, human-in-the-loop interaction proved highly effective: allowing humans to provide up to four short conversational hints more than doubled few-shot success on a subset of MBPP problems from 30% to 65%, cutting the overall error rate in half. In contrast, the models demonstrated severe deficits in program execution, failing to accurately predict the output of given programs and rarely exceeding 29% execution accuracy even when fine-tuned.

These findings indicate that large language models are powerful code-generation assistants when coupled with automated testing and human oversight, but they lack deep semantic understanding of program execution. Because a single natural language hint provided more performance gain than a five-fold increase in automated sample generations, the most effective practical application is collaborative pairing rather than fully autonomous programming. Furthermore, the risk of memorization is minimal, as training data overlap with the benchmark tasks was exceedingly low, but models remain prone to subtle logical errors on multi-step problems.

Organizations considering generative code tools should integrate these models as interactive developer assistants with automated test harnesses rather than autonomous systems. Leaders should establish testing pipelines that automatically validate generated code and design workflows enabling rapid human feedback. Before deploying such models into production engineering pipelines, teams must conduct additional analyses on code security, bias, and vulnerability risks.

Confidence in these findings is high for short, standalone programming functions under clear test conditions. However, caution is advised when extrapolating to complex software architectures. The benchmarks evaluated short, isolated Python scripts, and the models' inability to accurately trace program execution highlights fundamental limitations in their grasp of software semantics.

GoogleMassachusetts Institute of Technologyfew-shot learninggeneral purposelanguage modelModelCodeTest
arXiv: 2108.07732master/mbpp
  • Paper: Evaluating Large Language Models Trained on Code, Mark Chen et al. (2021). Evaluating Large Language Models Trained on Code established the Codex methodology and HumanEval benchmark that directly informed the code-synthesis evaluations and datasets analyzed in the source paper.
  • Paper: Textbooks Are All You Need, Suriya Gunasekar et al. (2023). Textbooks Are All You Need builds directly on the code-synthesis findings of the source paper by demonstrating that high-quality, targeted training data enables much smaller models to excel at Python generation.
Cover for Program Synthesis with Large Language Models

Abstract

This paper explores the limits of the current generation of large language models for program synthesis in general purpose programming languages. We evaluate a collection of such models (with between 244M and 137B parameters) on two new benchmarks, MBPP and MathQA-Python, in both the few-shot and fine-tuning regimes. Our benchmarks are designed to measure the ability of these models to synthesize short Python programs from natural language descriptions. The Mostly Basic Programming Problems (MBPP) dataset contains 974 programming tasks, designed to be solvable by entry-level programmers. The MathQA-Python dataset, a Python version of the MathQA benchmark, contains 23914 problems that evaluate the ability of the models to synthesize code from more complex text. On both datasets, we find that synthesis performance scales log-linearly with model size. Our largest models, even without finetuning on a code dataset, can synthesize solutions to 59.6 percent of the problems from MBPP using few-shot learning with a well-designed prompt. Fine-tuning on a held-out portion of the dataset improves performance by about 10 percentage points across most model sizes. On the MathQA-Python dataset, the largest fine-tuned model achieves 83.8 percent accuracy. Going further, we study the model's ability to engage in dialog about code, incorporating human feedback to improve its solutions. We find that natural language feedback from a human halves the error rate compared to the model's initial prediction. Additionally, we conduct an error analysis to shed light on where these models fall short and what types of programs are most difficult to generate. Finally, we explore the semantic grounding of these models by fine-tuning them to predict the results of program execution. We find that even our best models are generally unable to predict the output of a program given a specific input.

Topics

few-shot learninggeneral purposelanguage modellarge language modelsprogram executionprogramming languagespythontuning method

Keywords

ModelCodeTestModelsSynthesisProblemsTest CasesProgramLanguageDataset

Table of Contents

  • 1 Introduction
  • 2 Datasets
  • 2.1 Mostly Basic Programming Problems
  • 2.2 MathQA-Python
  • 3 Model and Methods
  • 4 MBPP Synthesis Results
  • 4.1 Synthesis Performance Improves as Model Size Increases
  • 4.2 Synthesis Performance is Insensitive to Number of Test Cases in Prompt
  • 4.3 Performance is Sensitive to Prompt Examples
  • 4.4 Solutions Typically Generalize to Held-Out Test Cases
  • 4.5 Programs Sometimes Overfit to Assert Statements
  • 4.6 Sampling Strategy is Critical for Good Performance
  • 4.7 Synthesis Performance Correlates Poorly with BLEU Score
  • 4.8 Pre-train / Test Overlap Seems to be Minimal
  • 4.9 Comparing Performance Between the Original and Edited Questions
  • 4.10 Qualitative Analysis of Error Modes
  • 5 Human-Model Collaboration Results
  • 5.1 Human Feedback Improves Synthesis Performance
  • 5.2 Qualitative Analysis of Human-Model Dialogs
  • 6 Program Execution Results
  • 6.1 Few-Shot Execution Performance is Poor
  • 6.2 Fine-tuning on Execution Slightly Improves Execution Performance
  • 6.3 Fine-tuning on Execution has a Small Effect on Synthesis Performance
  • 7 MathQA Results
  • 8 Related Work
  • 8.1 Machine Learning for Program Synthesis
  • 8.2 Machine Learning for Software Engineering
  • 8.3 Benchmarks for Machine Learning over Source Code
  • 9 Risks and Limitations
  • 10 Conclusion
  • References
  • A Appendix
  • A.1 Instructions given to crowd workers and expert reviewers
  • A.2 Instructions for human-model collaboration experiments
  • A.3 Prompts for execution experiments
  • A.4 Additional example human-model interaction samples

Knowls

  1. Knowl 1 — Mostly Basic Programming Problems (MBPP) Benchmark

    definition

    The Mostly Basic Programming Problems (MBPP) dataset is a benchmark designed to evaluate the capability of machine learning models to synthesize short Python programs from natural language specifications. It contains 974 programming tasks crowd-sourced from individuals with basic Python knowledge. Each task consists of:

    1. A short, natural language task description (typically one sentence).
    2. A single, self-contained Python function solving the problem.
    3. Three test cases written as Python assert statements that verify semantic and functional correctness.

    The ground-truth solutions average 6.8 lines of code (median 5, maximum 50). Problem types cover mathematical calculations (58%), list processing (43%), string processing (19%), integer sequences (9%), and other data structures (2%).

    A curated, hand-verified subset ("edited dataset") of 426 problems ensures standard Python function signatures, unambiguous descriptions, and accurate test cases. For standardized evaluation, MBPP is split into 10 held-out problems for few-shot prompting, 374 problems for fine-tuning, 90 for validation, and 500 problems for testing functional correctness.

  2. Knowl 2 — MathQA-Python Benchmark

    definition

    The MathQA-Python dataset is a benchmark for synthesizing Python code from complex mathematical word problem descriptions. It is constructed by converting the domain-specific language (DSL) operation sequences from the MathQA dataset into equivalent, executable Python code.

    To ensure semantic verification via code execution, the dataset is filtered to retain only problems whose program output precisely evaluates to the declared numerical answer, removing approximately 45% of the original MathQA dataset. The resulting dataset consists of 23,914 mathematical word problems partitioned into:

    • 19,209 training problems
    • 2,822 validation problems
    • 1,883 test problems

    Unlike benchmarks emphasizing complex control flow (loops and conditionals), MathQA-Python evaluates synthesis of largely straight-line algebraic and arithmetic operations from intricate natural language descriptions.

  3. Knowl 3 — Scaling and Performance of Decoder-Only Transformers on MBPP Program Synthesis

    empirical result

    Dense left-to-right autoregressive Transformer language models (ranging from 244 million to 137 billion non-embedding parameters, pre-trained on 2.81 trillion BPE tokens including 18.7 billion tokens from code-containing web documents) exhibit log-linear performance scaling with model parameter count on the MBPP benchmark.

    Evaluated by generating N=80N = 80 samples per problem at temperature T=0.5T = 0.5 and assessing functional correctness against 3 unit test assertions, synthesis success rates scale as follows:

    Model Size Few-Shot Tasks Solved (%) Fine-Tuned Tasks Solved (%)
    244M ≈1.5%\approx 1.5\% ≈8.0%\approx 8.0\%
    422M ≈3.0%\approx 3.0\% ≈14.0%\approx 14.0\%
    1B ≈6.0%\approx 6.0\% ≈23.0%\approx 23.0\%
    4B ≈18.0%\approx 18.0\% ≈31.0%\approx 31.0\%
    8B ≈30.0%\approx 30.0\% ≈41.0%\approx 41.0\%
    68B ≈39.0%\approx 39.0\% ≈54.0%\approx 54.0\%
    137B 59.6%59.6\% ≈69.5%\approx 69.5\%

    Fine-tuning on a small training split of 374 examples (100 training steps, learning rate 3×10−53 \times 10^{-5} for the 137B model) provides a consistent improvement of approximately 10 percentage points across most model sizes over few-shot prompting. On the curated 100-problem hand-verified MBPP subset, few-shot performance of the 137B model reaches 79.0% compared to 63.0% on the uncurated version.

  4. Knowl 4 — Program Synthesis Performance on MathQA-Python and MathQA-DSL

    data/table

    Evaluated on the 1,883 test problems of the MathQA dataset, large Transformer language models can synthesize solutions in both a domain-specific language (MathQA-DSL) and in Python (MathQA-Python). Correctness is measured by executing the generated program and checking if it yields the exact numerical answer.

    MathQA-DSL MathQA-Python
    Model Size Few-Shot (%) Fine-Tuned (%) Few-Shot (%) Fine-Tuned (%)
    8B 16.5% 79.0% 12.5% 74.7%
    68B 16.8% 82.8% 22.3% 79.5%
    137B 16.7% 83.8% 33.4% 81.2%

    In the few-shot regime (4 prompt examples), models perform substantially better on Python (up to 33.4% for 137B) than on DSL (16.7% for 137B) due to the presence of Python in pre-training corpora. After fine-tuning on 19,209 domain examples, DSL synthesis reaches higher peak accuracy (83.8% for 137B) than Python synthesis (81.2%), demonstrating that large language models can rapidly learn syntax for domain-specific languages given dedicated fine-tuning data.

  5. Knowl 5 — Multi-Turn Human-Model Dialog for Interactive Program Repair

    empirical result

    Allowing human programmers to provide short natural language feedback (at most one sentence containing hints, module imports, or identifier references, with up to 4 interactive turns) significantly improves program synthesis performance over single-turn generation.

    In an evaluation on 50 problems from the edited MBPP dataset using a 137B parameter language model primed with dialog examples:

    • Turn 0 (No human intervention, single sample): 30% of problems solved.
    • Turn 1 (Single feedback hint): 55% of problems solved.
    • Turn 4 (Up to four feedback hints): >65% of problems solved.

    A single natural language human hint reduces errors by 35.7% (from 70% failure to 45%), and 4 turns reduce errors by over 50% relative to initial prediction. Single-sample dialog repair at Turn 1 (55%) outperforms drawing 5 independent samples from the unassisted model without feedback (~53%), and human interaction enables solving 10 tasks that were completely unsolved by unassisted sampling.

  6. Knowl 6 — Program Execution Prediction and Semantic Grounding in Large Language Models

    empirical result

    When evaluated on predicting the execution output of Python programs (greedy decoding, temperature T=0.0T = 0.0), large language models (137B parameters) perform poorly, failing to demonstrate semantic simulation of program execution.

    On MBPP ground-truth programs evaluated across different prompt configurations:

    1 Test Case Output 2 Test Cases Output
    Prompt Information Included Few-Shot (%) Fine-Tuned (%) Few-Shot (%) Fine-Tuned (%)
    Source Code only 16.4% 20.8% 8.6% 9.0%
    Source Code + Natural Language 15.6% 20.6% 9.0% 8.2%
    Source Code + NL + I/O Examples 24.6% 23.2% 9.8% 8.4%
    Source Code + I/O Examples 28.8% 27.4% 11.6% 12.0%
    NL Description + I/O Examples 28.6% 28.2% 12.8% 13.0%
    NL Description only 17.6% 18.8% 8.4% 8.6%
    I/O Examples only 27.2% 26.2% 10.2% 13.0%

    Prompts containing input-output examples and natural language descriptions achieve higher execution accuracy (up to 28.8%) than prompts containing the ground-truth source code alone (16.4%), indicating that large language models infer outputs primarily by pattern matching from natural language and examples rather than by mentally tracing or executing Python code. Fine-tuning a 137B model on execution tasks yields a slight transfer benefit to program synthesis (+2.3% more correct samples and +3.6% more tasks solved).

  7. Knowl 7 — Insensitivity of Code Synthesis to Number of Prompt Assertions

    empirical result

    In few-shot Python code synthesis using a 137B parameter language model on the MBPP dataset, varying the number of input-output test assertions provided in the prompt does not substantially alter the problem solve rate:

    Number of Prompt Assertions Tasks Solved (%) Samples Correct (%)
    0 (function name only) 43.2% 10.23%
    1 55.2% 15.30%
    2 59.0% 15.14%
    3 58.4% 16.77%

    Providing at least one example assertion improves task success from 43.2% to 55.2% primarily by anchoring function signature conventions and types, but increasing from 1 to 3 assertions yields negligible additional task-level gains (55.2% vs 58.4%). This demonstrates that the model does not systematically use additional test cases to reason about program semantics.

  8. Knowl 8 — Prompt Example Sensitivity and Prompt Ensembling in Few-Shot Synthesis

    empirical result

    Few-shot program synthesis performance in large language models is highly sensitive to the specific choice of exemplar tasks included in the prompt context.

    Across 15 random seeds used to select held-out few-shot exemplars on the MBPP dataset for a 137B parameter model, task solve rates varied from under 10% to approximately 60%. Prompts containing long, verbose examples often induce repetitive failure loops or exceed context window limits (512 or 1024 tokens), whereas short, compact exemplars utilizing standard libraries yield higher performance.

    Furthermore, different prompt seeds solve non-overlapping subsets of problems. Taking the union of solved tasks across prompt seeds (prompt ensembling) increases the overall percentage of MBPP problems solved by the 137B model from 59.6% (single best seed) to 66.4%.

  9. Knowl 9 — Generalization vs Overfitting to Assert Statements in LLM Code Synthesis

    empirical result

    Programs synthesized by large language models on the MBPP benchmark generally generalize beyond the unit test assertions provided in the prompt rather than overfitting or trivializing them:

    1. Adversarial generalization test: In a sample of 50 MBPP test tasks evaluated against newly constructed challenge/adversarial unit tests designed to catch shortcut implementations (e.g., handling strings with multiple target characters rather than just two), 66% (33/50) passed the original assertions and 58% (29/50) passed both original and challenge assertions. This indicates that approximately 88% (2933\frac{29}{33}) of solved programs implement true generalized logic, with only ≈12%\approx 12\% failing edge-case verification.
    2. Hard-coding shortcuts: In rare instances, models exhibit shortcut overfitting by directly parsing the assertion literals in the prompt and generating trivial condition trees (e.g., if n == 383: return True else: return False when only 383 is asserted True for a sequence check). However, such occurrences are infrequent across the full dataset.

  10. Knowl 10 — Sampling Temperature, Beam Search, and Metric Disconnect in Code Generation

    empirical result

    The choice of decoding strategy and evaluation metrics strongly affects the assessment of program synthesis with large language models:

    • Temperature sampling vs. Beam search: Greedy decoding (temperature T=0.1T = 0.1) achieves higher single-sample success rates on MBPP, but higher temperatures (T=0.5T = 0.5 to T=0.8T = 0.8) scale far better when a budget of N≥10N \ge 10 samples is drawn (reaching peak pass rates at N=80N = 80). Beam search performs worse than all temperature sampling configurations due to degenerate infinite loops and repeated lines of code.
    • BLEU score disconnect: BLEU score between generated code and ground-truth reference implementations exhibits virtually no correlation with functional correctness (test execution success). Semantically correct solutions frequently achieve low n-gram overlap due to identifier renaming, alternative algorithms, and variable naming differences.

  11. Knowl 11 — Qualitative Error Modes in Large Language Model Code Synthesis

    empirical result

    Qualitative analysis of failure cases on the MBPP benchmark reveals three primary failure modes in large language model program synthesis:

    1. Multi-constraint and compound sub-problems: The model frequently produces partial solutions that address only one sub-component of a multi-part task (e.g., finding identical characters but failing to compute the longest palindromic subsequence; or counting binary digits without searching over all substrings).
    2. Linguistic off-by-one / common sibling errors: When a task description shares prominent keywords with common programming problems (e.g., "find the largest number that can be formed from digits"), the model frequently solves the more prevalent sibling problem (e.g., finding the maximum single digit in the list).
    3. Context drift in interactive dialog: During multi-turn interactive repair, models frequently struggle to retain initial function signatures or revert incorrect intermediate edits when instructed to return to earlier states.

  12. Knowl 12 — Pre-Training Data Overlap Analysis for MBPP

    empirical result

    To verify that program synthesis results on the MBPP dataset reflect generalization rather than verbatim memorization of web pre-training data, an exact line-level overlap analysis was performed across the 2.97 billion document pre-training corpus:

    • After stripping leading/trailing whitespace and excluding common Python syntax lines occurring more than twice across MBPP (such as return or continue), 91.5% of the 974 MBPP ground-truth programs had at most 1 or 2 lines matching any document in the pre-training set.
    • Only 32 out of 974 problems (3.3%) had more than 50% of their lines matched in the pre-training corpus.

Coverage note — None was omitted; all major datasets, quantitative scaling curves, dialog interaction experiments, execution evaluations, error breakdowns, and memorization analyses are represented.

References

  1. 1.Ferran Alet, Javier Lopez-Contreras, James Koppel, Maxwell Nye, Armando Solar-Lezama, Tomas Lozano-Perez, Leslie Kaelbling, and Joshua Tenenbaum. A large-scale benchmark for few-shot program induction and synthesis. In Marina Meila and Tong Zhang, editors, International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pages 175–186. PMLR, 18–24 Jul 2021. URL http://proceedings.mlr.press/v139/alet21a.html.
  2. 2.Miltiadis Allamanis. A survey of machine learning on source code. https://ml4code.github.io/, 2021. Accessed 2021-07-11.
  3. 3.Miltiadis Allamanis, Earl T Barr, Christian Bird, and Charles Sutton. Learning natural coding conventions. In Symposium on the Foundations of Software Engineering (FSE), 2014.
  4. 4.Miltiadis Allamanis, Hao Peng, and Charles Sutton. A Convolutional Attention Network for Extreme Summarization of Source Code. In International Conference in Machine Learning (ICML), 2016.
  5. 5.Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. A survey of machine learning for big code and naturalness. ACM Computing Surveys, 51(4), September 2018a.
  6. 6.Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. Learning to represent programs with graphs. In International Conference on Learning Representations (ICLR), February 2018b.
  7. 7.Miltos Allamanis and Charles Sutton. Mining source code repositories at massive scale using language modeling. In Working Conference on Mining Software Repositories (MSR), 2013a.
  8. 8.Miltos Allamanis and Charles Sutton. Mining source code repositories at massive scale using language modeling. In Working Conference on Mining Software Repositories (MSR), 2013b.
  9. 9.Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. code2vec: Learning distributed representations of code. Proceedings of the ACM on Programming Languages, 3(POPL):40, 2019.
  10. 10.Rajeev Alur, Rastislav Bodík, Garvit Juniwal, Milo M. K. Martin, Mukund Raghothaman, Sanjit A. Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. Syntax-guided synthesis. In Formal Methods in Computer-Aided Design, FMCAD 2013, Portland, OR, USA, October 20-23, 2013, pages 1–8. IEEE, 2013. URL http://ieeexplore.ieee.org/document/6679385/.
  11. 11.Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. CoRR, abs/1905.13319, 2019. URL http://arxiv.org/abs/1905.13319.
  12. 12.J. W. Backus, R. J. Beeber, S. Best, R. Goldberg, L. M. Haibt, H. L. Herrick, R. A. Nelson, D. Sayre, P. B. Sheridan, H. Stern, I. Ziller, R. A. Hughes, and R. Nutt. The FORTRAN automatic coding system. In Papers Presented at the February 26-28, 1957, Western Joint Computer Conference: Techniques for Reliability, IRE-AIEE-ACM ’57 (Western), pages 188–198, New York, NY, USA, 1957. ACM. doi: 10.1145/1455567.1455599. URL http://doi.acm.org/10.1145/1455567.1455599.
  13. 13.Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. Deepcoder: Learning to write programs. In International Conference on Learning Representations (ICLR), 2017. arXiv:1611.01989.
  14. 14.Emily M. Bender and Alexander Koller. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.463. URL https://aclanthology.org/2020.acl-main.463.
  15. 15.Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, pages 610–623, New York, NY, USA, March 2021. Association for Computing Machinery.
  16. 16.David Bieber, Charles Sutton, Hugo Larochelle, and Daniel Tarlow. Learning to execute programs with instruction pointer attention graph neural networks. In H. Larochelle, M. Ranzato, H. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 8626–8637. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/62326dc7c4f7b849d6f013ba46489d6c-Paper.pdf.
  17. 17.big-bench collaboration. Beyond the imitation game: Measuring and extrapolating the capabilities of language models. In preparation, 2021. URL https://github.com/google/BIG-bench/.
  18. 18.Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large scale autoregressive language modeling with mesh-tensorflow, 2021. URL http://github.com/eleutherai/gpt-neo.
  19. 19.Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/2005.14165, 2020. URL https://arxiv.org/abs/2005.14165.
  20. 20.Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. arXiv preprint arXiv:2012.07805, 2020.
  21. 21.Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, Will Guss, Alex Nichol, Igor Babuschkin, Suchir Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, July 2021. URL http://arxiv.org/abs/2107.03374.
  22. 22.Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019a. URL https://openreview.net/forum?id=H1gfOiAqYm.
  23. 23.Zimin Chen, S J Kommrusch, M Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus. Sequencer: Sequence-to-sequence learning for end-to-end program repair. IEEE Transactions on Software Engineering, 2019b.
  24. 24.Colin B. Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. PyMT5: Multi-mode translation of natural language and python code with transformers. 2020. URL http://arxiv.org/abs/2010.03150.
  25. 25.B.J. Copeland. Alan Turing’s Electronic Brain: The Struggle to Build the ACE, the World’s Fastest Computer. OUP Oxford, 2012. ISBN 9780199609154. URL https://books.google.com/books?id=YhQZnczOS7kC.
  26. 26.Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Neural Information Processing Systems, 2015.
  27. 27.Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy I/O. CoRR, abs/1703.07469, 2017. URL http://arxiv.org/abs/1703.07469.
  28. 28.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019.
  29. 29.Elizabeth Dinella, Hanjun Dai, Ziyang Li, Mayur Naik, Le Song, and Ke Wang. Hoppity: Learning graph transformations to detect and fix bugs in programs. In International Conference on Learning Representations, September 2019.
  30. 30.Kevin Ellis, Lucas Morales, Mathias Sablé-Meyer, Armando Solar-Lezama, and Josh Tenenbaum. Learning libraries of subroutines for neurally–guided bayesian program induction. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 7805–7815. Curran Associates, Inc., 2018.
  31. 31.Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar-Lezama. Write, execute, assess: Program synthesis with a REPL. In NeurIPS, 2019.
  32. 32.Kevin Ellis, Catherine Wong, Maxwell I. Nye, Mathias Sablé-Meyer, Luc Cary, Lucas Morales, Luke B. Hewitt, Armando Solar-Lezama, and Joshua B. Tenenbaum. Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleep bayesian program learning. CoRR, abs/2006.08381, 2020. URL https://arxiv.org/abs/2006.08381.
  33. 33.Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A pre-trained model for programming and natural languages. Findings of the Association for Computational Linguistics: EMNLP 2020, February 2020.
  34. 34.Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. CoRR, abs/1410.5401, 2014.
  35. 35.Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwinska, Sergio Gomez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471–476, 2016.
  36. 36.Sumit Gulwani. Automating string processing in spreadsheets using input-output examples. In Proceedings of the 38th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL ’11, pages 317–330, New York, NY, USA, 2011. ACM. ISBN 978-1-4503-0490-0. doi: 10.1145/1926385.1926423. URL http://doi.acm.org/10.1145/1926385.1926423.
  37. 37.Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, et al. Program synthesis. Foundations and Trends® in Programming Languages, 4(1-2):1–119, 2017.
  38. 38.Vincent J Hellendoorn, Christian Bird, Earl T Barr, and Miltiadis Allamanis. Deep learning type inference. In ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 152–162, New York, New York, USA, 2018. ACM Press.
  39. 39.Vincent J Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. Global relational models of source code. In International Conference on Learning Representations (ICLR), September 2019.
  40. 40.Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. May 2021. URL http://arxiv.org/abs/2105.09938.
  41. 41.Abram Hindle, Earl Barr, Zhendong Su, Prem Devanbu, and Mark Gable. On the “naturalness” of software. In International Conference on Software Engineering (ICSE). 2012.
  42. 42.Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Association of Computational Linguistics (ACL), 2018.
  43. 43.Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. CodeSearchNet challenge: Evaluating the state of semantic code search. September 2019. URL http://arxiv.org/abs/1909.09436.
  44. 44.Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language to code in programmatic context. In Empirical Methods in Natural Language Processing (EMNLP), August 2018.
  45. 45.Ellen Jiang, Edwin Toh, Alejandra Molina, Aaron Donsbach, Carrie Cai, and Michael Terry. Genline and genform: Two tools for interacting with generative language models in a code editor. Adjunct Publication of the 34th Annual ACM Symposium on User Interface Software and Technology, 2021.
  46. 46.Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.
  47. 47.Lukasz Kaiser and Ilya Sutskever. Neural gpus learn algorithms. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
  48. 48.Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning (ICML), 2020.
  49. 49.Svetoslav Karaivanov, Veselin Raychev, and Martin Vechev. Phrase-Based statistical translation of programming languages. In Onward, 2014.
  50. 50.Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, and Andrea Janes. Big code != big vocabulary: Open-Vocabulary models for source code. In International Conference on Software Engineering (ICSE), March 2020.
  51. 51.Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. Code prediction by feeding trees to transformers. March 2020. URL http://arxiv.org/abs/2003.13848.
  52. 52.Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 - November 4, 2018, pages 66–71. Association for Computational Linguistics, 2018. doi: 10.18653/v1/d18-2012. URL https://doi.org/10.18653/v1/d18-2012.
  53. 53.Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. SPoC: Search-based pseudocode to code. In Advances in Neural Information Processing Systems, 2019.
  54. 54.Karol Kurach, Marcin Andrychowicz, and Ilya Sutskever. Neural random-access machines. In International Conference on Learning Representations, (ICLR), 2016.
  55. 55.Claire Le Goues, Thanhvu Nguyen, Stephanie Forrest, and Westley Weimer. GenProg: A generic method for automatic software repair. IEEE Trans. Software Eng., 38(1):54–72, January 2012.
  56. 56.Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. April 2021. URL http://arxiv.org/abs/2104.08691.
  57. 57.Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. ArXiv, abs/2106.00737, 2021.
  58. 58.Xiang Lisa Li and Percy Liang. Prefix-Tuning: Optimizing continuous prompts for generation. January 2021. URL http://arxiv.org/abs/2101.00190.
  59. 59.Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomás Kociský, Fumin Wang, and Andrew Senior. Latent predictor networks for code generation. In Annual Meeting of the Association for Computational Linguistics (ACL), 2016.
  60. 60.Fan Long and Martin Rinard. Automatic patch generation by learning correct code. In ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL), pages 298–312, New York, NY, USA, January 2016. Association for Computing Machinery.
  61. 61.Annie Louis, Santanu Kumar Dash, Earl T Barr, Michael D Ernst, and Charles Sutton. Where should I comment my code? A dataset and model for predicting locations that need comments. In International Conference on Software Engineering (ICSE; NIER track), 2020.
  62. 62.Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. February 2021. URL http://arxiv.org/abs/2102.04664.
  63. 63.Chris J Maddison and Daniel Tarlow. Structured generative models of natural source code. In International Conference on Machine Learning (ICML), pages 649–657. 2014.
  64. 64.Zohar Manna and Richard Waldinger. Knowledge and reasoning in program synthesis. Artificial Intelligence, 6(2):175–208, 1975.
  65. 65.Zohar Manna and Richard J Waldinger. Toward automatic program synthesis. Communications of the ACM, 14(3):151–165, 1971.
  66. 66.Antonio Mastropaolo, Simone Scalabrino, Nathan Cooper, David Nader Palacio, Denys Poshyvanyk, Rocco Oliveto, and Gabriele Bavota. Studying the usage of Text-To-Text transfer transformer to support Code-Related tasks. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pages 336–347, May 2021.
  67. 67.Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. Recurrent neural network ` based language model. In INTERSPEECH, 2010.
  68. 68.Vijayaraghavan Murali, Letao Qi, Swarat Chaudhuri, and Chris Jermaine. Neural sketch learning for conditional program generation. In International Conference on Learning Representations (ICLR), 2018.
  69. 69.Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. Lexical statistical machine translation for language migration. In Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE), NIER, 2013.
  70. 70.Augustus Odena and Charles Sutton. Learning to represent programs with property signatures. CoRR, abs/2002.09030, 2020. URL https://arxiv.org/abs/2002.09030.
  71. 71.Augustus Odena, Kensen Shi, David Bieber, Rishabh Singh, and Charles Sutton. BUSTLE: bottom-up program synthesis through learning-guided exploration. CoRR, abs/2007.14381, 2020. URL https://arxiv.org/abs/2007.14381.
  72. 72.Irene Vlassi Pandi, Earl T Barr, Andrew D Gordon, and Charles Sutton. OptTyper: Probabilistic type inference by optimising logical and natural constraints. April 2020. URL http://arxiv.org/abs/2004.00348.
  73. 73.Sheena Panthaplackel, Junyi Jessy Li, Milos Gligoric, and Raymond J Mooney. Deep Just-In-Time inconsistency detection between comments and source code. In AAAI Conference on Artificial Intelligence, 2021.
  74. 74.Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In North American Chapter of the Association for Computational Linguistics (NAACL), 2018.
  75. 75.Amir Pnueli and Roni Rosner. On the synthesis of a reactive module. In SIGPLAN-SIGACT symposium on Principles of Programming Languages (POPL), pages 179–190. ACM, 1989.
  76. 76.Michael Pradel and Koushik Sen. DeepBugs: a learning approach to name-based bug detection. Proc. ACM Program. Lang., 2(OOPSLA):1–25, October 2018.
  77. 77.Michael Pradel, Georgios Gousios, Jason Liu, and Satish Chandra. TypeWriter: neural type prediction with search-based validation. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020, pages 209–220, New York, NY, USA, November 2020. Association for Computing Machinery.
  78. 78.Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladmir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, and Ulrich Finkler. Project CodeNet: A Large-Scale AI for code dataset for learning a diversity of coding tasks. May 2021. URL http://arxiv.org/abs/2105.12655.
  79. 79.Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. https://blog.openai.com/language-unsupervised, 2018.
  80. 80.Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
  81. 81.Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683, 2019. URL http://arxiv.org/abs/1910.10683.
  82. 82.Veselin Raychev, Martin Vechev, and Eran Yahav. Code completion with statistical language models. In ACM PLDI, 2014.
  83. 83.Veselin Raychev, Martin Vechev, and Andreas Krause. Predicting program properties from “big code”. In ACM Symposium on Principles of Programming Languages (POPL), 2015.
  84. 84.Veselin Raychev, Pavol Bielik, and Martin Vechev. Probabilistic model for code with decision trees. In OOPSLA, 2016.
  85. 85.Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages. Adv. Neural Inf. Process. Syst., 33:20601–20611, 2020.
  86. 86.Roei Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov. You autocomplete me: Poisoning vulnerabilities in neural code completion. In 30th USENIX Security Symposium (USENIX Security 21), July 2020.
  87. 87.Tal Schuster, Ashwin Kalyan, Oleksandr Polozov, and Adam Tauman Kalai. Programming puzzles. June 2021. URL http://arxiv.org/abs/2106.05784.
  88. 88.David E. Shaw, William R. Swartout, and C. Cordell Green. Inferring LISP programs from examples. In International Joint Conference on Artificial Intelligence (IJCAI), pages 260–267, 1975. URL http://ijcai.org/Proceedings/75/Papers/037.pdf.
  89. 89.Armando Solar-Lezama. Introduction to program synthesis. https://people.csail.mit.edu/asolar/SynthesisCourse/TOC.htm, 2018. Accessed: 2018-09-17.
  90. 90.Armando Solar-Lezama, Liviu Tancau, Rastislav Bodík, Sanjit A. Seshia, and Vijay A. Saraswat. Combinatorial sketching for finite programs. In Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2006, San Jose, CA, USA, October 21-25, 2006, pages 404–415. ACM, 2006.
  91. 91.Phillip D Summers. A methodology for LISP program construction from examples. Journal of the ACM (JACM), 24(1):161–175, 1977.
  92. 92.Ilya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural networks. In International Conference on Machine Learning (ICML), 2011.
  93. 93.Jeffrey Svajlenko, Judith F Islam, Iman Keivanloo, Chanchal K Roy, and Mohammad Mamun Mia. Towards a big data curated benchmark of inter-project code clones. In 2014 IEEE International Conference on Software Maintenance and Evolution, pages 476–480, September 2014.
  94. 94.Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. IntelliCode compose: Code generation using transformer. May 2020. URL http://arxiv.org/abs/2005.08025.
  95. 95.Daniel Tarlow, Subhodeep Moitra, Andrew Rice, Zimin Chen, Pierre-Antoine Manzagol, Charles Sutton, and Edward Aftandilian. Learning to fix build errors with Graph2Diff neural networks. November 2019. URL http://arxiv.org/abs/1911.01205.
  96. 96.Emina Torlak and Rastislav Bodik. Growing solver-aided languages with rosette. In ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software, Onward! 2013, pages 135–152, New York, NY, USA, October 2013. Association for Computing Machinery.
  97. 97.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
  98. 98.R.J. Waldinger, R.C.T. Lee, and SRI International. PROW: A Step Toward Automatic Program Writing. SRI International, 1969. URL https://books.google.com/books?id=3BITSQAACAAJ.
  99. 99.Jiayi Wei, Maruth Goyal, Greg Durrett, and Isil Dillig. LambdaNet: Probabilistic type inference using graph neural networks. In International Conference on Learning Representations, 2020.
  100. 100.Michihiro Yasunaga and Percy Liang. Graph-based, self-supervised program repair from diagnostic feedback. In International Conference on Machine Learning, May 2020.
  101. 101.Pengcheng Yin and Graham Neubig. A syntactic neural model for general-purpose code generation. In Association for Computational Linguistics (ACL), 2017.
  102. 102.Wojciech Zaremba and Ilya Sutskever. Learning to execute. ArXiv, abs/1410.4615, 2014.
  103. 103.Maksym Zavershynskyi, Alex Skidanov, and Illia Polosukhin. NAPS: Natural program synthesis dataset. In Workshop on Neural Abstract Machines & Program Induction (NAMPI), July 2018.
  104. 104.Amit Zohar and Lior Wolf. Automatic program synthesis of long programs with a learned garbage collector. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 2094–2103. Curran Associates, Inc., 2018.

Citation

MLA
Austin, J., et al. “Program Synthesis with Large Language Models”. arXiv, 2021, http://arxiv.org/abs/2108.07732v1.
APA
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., & Sutton, C. (2021). Program Synthesis with Large Language Models. arXiv. http://arxiv.org/abs/2108.07732v1
Chicago
Austin, J., A. Odena, M. Nye, et al. 2021. “Program Synthesis with Large Language Models”. arXiv. http://arxiv.org/abs/2108.07732v1.
Harvard
Austin, J. et al. (2021) “Program Synthesis with Large Language Models”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2108.07732v1.
Vancouver
1. Austin J, Odena A, Nye M, et al (2021) Program Synthesis with Large Language Models. arXiv

BibTeX

@article{austin2021program,
  title = {Program Synthesis with Large Language Models},
  author = {Austin, Jacob and Odena, Augustus and Nye, Maxwell and Bosma, Maarten and Michalewski, Henryk and Dohan, David and Jiang, Ellen and Cai, Carrie and Terry, Michael and Le, Quoc and Sutton, Charles},
  year = {2021},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2108.07732v1},
  eprint = {2108.07732}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Published with permission