Back

Competition-level code generation with AlphaCode

Yujia LiDavid ChoiJunyoung ChungNate KushmanJulian SchrittwieserR. LeblondTomEcclesJames KeelingFelix Gimeno

article2022Science2,511 citations

Presents AlphaCode, a code generation system that solves complex algorithmic challenges and achieves top 54.3% performance in human programming competitions on Codeforces by combining large-scale model sampling with execution-based filtering.

Listen

Developing artificial intelligence systems capable of writing software independently has long been a major goal in computer science. While recent large-scale language models can solve simple coding tasks or generate short code snippets, they consistently struggle with complex, unseen problems that require deeper algorithmic reasoning, mathematical problem-solving, and handling complex natural language specifications.

The article sets out to develop and evaluate AlphaCode, a code-generation system capable of creating novel solutions to challenging competitive programming problems at a level comparable to human competitors.

To evaluate this capability credibly, the authors introduced CodeContests, a curated dataset of competitive programming tasks featuring a strict temporal split to prevent data leakage and extensive generated test cases that reduced evaluation false positive rates from around 30–60% down to 4%. AlphaCode utilizes an asymmetric encoder-decoder transformer architecture pre-trained on 715 GB of open-source GitHub code and fine-tuned on CodeContests using offline reinforcement learning and regularization techniques. The system searches the program space by generating a massive volume of candidate solutions per problem (up to one million), filtering them down using public example test cases, and clustering the remaining candidates by their execution behavior to select a budget of at most 10 final submissions.

The investigation produced several key findings. First, in simulated evaluations across 10 recent programming competitions on the Codeforces platform (each with over 5,000 human participants), AlphaCode achieved an average ranking in the top 54.3%, corresponding to an estimated rating of 1238—placing it in the top 28% of active human competitors. Second, on the CodeContests dataset, the best system solved 34.2% of unseen problems using 10 submissions chosen from one million samples, significantly exceeding previous models that typically achieved low single-digit solve rates. Third, solve rates scaled log-linearly with model parameter scale, dataset size, training compute, and the number of drawn samples. Fourth, detailed analysis confirmed that the system does not simply copy code blocks or exploit dataset flaws, but actively reasons over the natural language task descriptions to generate original solutions.

These findings demonstrate for the first time that an automated system can reach human-median performance on open-ended, algorithmic programming benchmarks. In practice, this marks a substantial step toward automating software engineering workflows, improving developer productivity, and expanding access to programming education. However, the reliance on generating millions of samples introduces notable computational costs and energy demands, and automated code generation introduces potential security risks such as the creation of exploitable or malicious code.

For future development, organizations looking to build or deploy code generation systems should prioritize sampling efficiency and architectural improvements, such as multi-query attention, alongside behavioral filtering to manage computational expenses. Further work is recommended to design training objectives that align more closely with solve rates rather than standard loss metrics, as well as to establish robust safety, licensing, and vulnerability checks for generated code.

Readers should note specific limitations: AlphaCode struggles with certain problem classes, such as dynamic programming, and its strong performance relies heavily on large-scale parallel sampling and the availability of test cases for filtering. While confidence in the benchmarked competition results is high due to rigorous temporal splitting and verified test coverage, running such systems in production or live environments remains constrained by compute budgets and the need to handle edge-case behavior.

Googlecode generationencoder-decoderlanguage modelModelCodeTest
arXiv: 2203.07814google-deepmind/code_contests
Cover for Competition-level code generation with AlphaCode

Abstract

Programming is a powerful and ubiquitous problem-solving tool. Developing systems that can assist programmers or even generate programs independently could make programming more productive and accessible, yet so far incorporating innovations in AI has proven challenging. Recent large-scale language models have demonstrated an impressive ability to generate code, and are now able to complete simple programming tasks. However, these models still perform poorly when evaluated on more complex, unseen problems that require problem-solving skills beyond simply translating instructions into code. For example, competitive programming problems which require an understanding of algorithms and complex natural language remain extremely challenging. To address this gap, we introduce AlphaCode, a system for code generation that can create novel solutions to these problems that require deeper reasoning. In simulated evaluations on recent programming competitions on the Codeforces platform, AlphaCode achieved on average a ranking of top 54.3% in competitions with more than 5,000 participants. We found that three key components were critical to achieve good and reliable performance: (1) an extensive and clean competitive programming dataset for training and evaluation, (2) large and efficient-to-sample transformer-based architectures, and (3) large-scale model sampling to explore the search space, followed by filtering based on program behavior to a small set of submissions.

Topics

code generationencoder-decoderlanguage modelnatural languagesprogramming languagesreasoning

Keywords

ModelCodeTestCompetition-level Code GenerationProblemsModelsSolveAlphaCodeSolve RateSamples

Table of Contents

  • 1 Introduction
  • 2 Problem setup
  • 2.1 Competitive programming
  • 2.2 Evaluation
  • 3 Datasets
  • 3.1 Pre-training dataset
  • 3.2 CodeContests fine-tuning dataset
  • 3.2.1 False positives and additional generated tests
  • 4 Approach
  • 4.1 Model architecture
  • 4.2 Pre-training
  • 4.3 Fine-tuning
  • 4.4 Large scale sampling
  • 4.5 Filtering
  • 4.6 Clustering
  • 5 Results
  • 5.1 Codeforces competitions evaluation
  • 5.2 CodeContests evaluation
  • 5.3 CodeContests ablations & results
  • 5.3.1 Solve rates scale with respect to parameter count, compute, number of samples, and dataset size
  • 5.3.2 Architecture changes to improve sampling speed
  • 5.3.3 Choice of the pre-training dataset
  • 5.3.4 Model enhancements
  • 5.3.5 Filtering & clustering
  • 5.4 Results on APPS
  • 6 AlphaCode’s capabilities & limitations
  • 6.1 Copying from training data
  • 6.2 Model solution characteristics
  • 6.3 Sensitivity to problem descriptions
  • 6.4 Sensitivity to provided metadata
  • 6.5 Loss is a poor proxy for solve rate
  • 7 Related work
  • 7.1 Program synthesis
  • 7.2 Transformers for program synthesis
  • 7.3 Scaling sampling
  • 7.4 Evaluation metrics
  • 7.5 Competitive programming
  • 8 Broader impact
  • 8.1 Applications
  • 8.2 Potential risks and benefits
  • 9 Conclusion
  • References
  • 10 Appendix
  • A Problem setup
  • A.1 Hidden tests
  • A.2 Program judging
  • A.3 Evaluation metrics
  • B Datasets
  • B.1 GitHub dataset composition
  • B.2 Dataset cleaning
  • B.3 Data leakage and temporal split
  • C Approach and Results
  • C.1 Ensembling
  • C.2 Metadata conditioning
  • C.3 GOLD
  • C.4 Additional results for filtering and clustering
  • C.5 HumanEval comparison
  • C.6 APPS dataset settings
  • C.7 Best settings for sampling
  • C.8 Scaling with dataset size
  • D Codeforces contest evaluation
  • D.1 Simulation
  • D.2 Multiple evaluations
  • D.3 Results
  • E Additional analysis of AlphaCode’s capabilities and limitations
  • E.1 Model sample statistics
  • E.2 Solve rate for different problem difficulty ratings
  • E.3 Sensitivity to the problem descriptions
  • E.3.1 Simplification of problem descriptions
  • E.3.2 Incorrect or irrelevant rewordings
  • E.3.3 Capturing variables and their relations
  • E.3.4 Sensitivity to word-level changes
  • E.3.5 Description section ablations
  • E.4 Sensitivity to problem metadata
  • E.4.1 Problem ratings
  • E.4.2 Solution correctness
  • F Complete prompt and model examples
  • F.1 Solution duplication
  • F.1.1 Solution decompositions
  • F.1.2 Very long common subsequences between human solutions and finetuning data
  • F.2 Problem description rewordings
  • F.2.1 Simplified rewordings
  • F.2.2 Incorrect and verbose rewordings

Knowls

  1. Knowl 1 — AlphaCode End-to-End System Pipeline

    model/method

    AlphaCode is an end-to-end system for generating competitive programming solutions directly from natural language problem descriptions. The pipeline operates in five stages:

    1. Large-Scale Pre-training: An encoder-decoder transformer is pre-trained on 715.1 GB of public GitHub source code across 12 programming languages (C++, C#, Go, Java, JavaScript, Lua, PHP, Python, Ruby, Rust, Scala, TypeScript) using causal next-token prediction on the decoder and masked language modeling (MLM) on the encoder.
    2. Fine-Tuning on Competitive Programming: The model is fine-tuned on the CodeContests dataset using natural language descriptions as encoder input and solution code as decoder target. Fine-tuning incorporates masked language modeling, softmax tempering (T=0.2T=0.2), metadata conditioning, value conditioning, and off-policy reinforcement learning via GOLD.
    3. Massive Parallel Sampling: For each unseen competition problem, the fine-tuned model generates a large budget of candidate solutions (from 10310^3 up to 10610^6 samples) using temperature sampling with randomized conditioning metadata (ratings, tags, and programming language: half Python, half C++).
    4. Example-Test Filtering: Candidate programs are executed against the public example input/output test cases included in the problem description. Programs that fail to compile or do not produce the expected example outputs are discarded, filtering out approximately 99% of generated candidates.
    5. Semantic Clustering and Submission Selection: A separate learned input-generation model produces synthetic test inputs. The remaining filtered candidates are executed on these inputs and clustered according to their execution output signatures. One candidate is selected from each cluster in order of descending cluster size to produce a small submission budget (e.g., n≤10n \le 10).

  2. Knowl 2 — CodeContests Dataset and Test Case Generation

    experimental setup

    CodeContests is a competitive programming dataset curated to evaluate and train code generation models while addressing data leakage and test under-specification.

    • Dataset Composition and Temporal Split: The dataset comprises 13,328 training problems, 117 validation problems, and 165 test problems, scraped from Codeforces and augmented with public competitive programming datasets (Description2Code and CodeNet). To guarantee that evaluation problems are strictly unseen, a temporal split is enforced: all pre-training and fine-tuning training data appeared publicly on or before 2021/07/14; validation problems appeared between 2021/07/15 and 2021/09/20; test problems appeared after 2021/09/21.
    • Problem Contents: Each problem includes a natural language description, input/output format specifications, public example tests, hidden platform tests, difficulty ratings (800–3500), algorithm tags, and human submissions in C++, Python, and Java (both correct and incorrect).
    • Synthetic Test Generation for False Positive Reduction: Because existing benchmarks suffer from high false positive rates due to small test suites (60% on APPS, 30% on HumanEval, 62% on raw Codeforces scrapes), additional test inputs are generated by mutating existing test inputs (bit flips, integer increments/decrements, string character perturbations). Mutated test inputs are verified by executing 30 known correct human solutions and retaining only inputs where all 30 solutions produce identical output (up to 200 generated tests per problem). Problems with fewer than 5 tests or with trivial constant outputs are filtered out. This process reduces the evaluation false positive rate to 4.0%.

  3. Knowl 3 — The n@k Evaluation Metric and Hypergeometric Bootstrap Estimation

    algorithm

    To evaluate code generation under the realistic constraints of programming competitions—where candidates can be sampled repeatedly but only a small submission budget nn can be submitted to hidden tests without penalty—the paper defines the n@kn@k metric. The quantity n@kn@k denotes the percentage of problems solved when generating kk candidate samples per problem, filtering them using public example tests, and submitting at most n≤kn \le k of the filtered candidates for evaluation against hidden tests. When n=kn=k, the metric equals pass@kpass@k.

    To compute n@kn@k with low variance from a large sample pool K≫kK \gg k drawn from a model, an unbiased estimator uses hypergeometric sampling over SS subsamples:

    Input: n (number of allowed submissions per problem)
    Input: k (number of allowed samples per problem)
    Input: e_p (number of samples passing public example tests for problem p)
    Input: s_p (number of samples passing all hidden tests for problem p)
    Input: K (total number of samples evaluated for problem p, where K >= k)
    Input: S (number of Monte Carlo subsamples)
    Output: Estimated n@k solve rate across all problems
    for each problem p in the problem set do
        for each subsample i from 1 to S do
            Sample e'_p ~ Hypergeometric(e_p, K - e_p, k)
            n' = min(e'_p, n)
            Sample s'_p ~ Hypergeometric(s_p, e_p - s_p, n')
            if s'_p > 0 then
                solved_{p, i} = 1
            else
                solved_{p, i} = 0
            end if
        end for
        Compute solve_rate_p as the average of solved_{p, i} over all S subsamples
    end for
    return the average solve_rate_p across all problems in the problem set

  4. Knowl 4 — Asymmetric Transformer Architecture with Multi-Query Attention

    model/method

    AlphaCode uses an encoder-decoder transformer architecture with architectural adaptations tailored for long natural language prompts and fast autoregressive code generation:

    • Asymmetric Sequence Lengths: Problem descriptions are typically longer than solutions. The architecture allocates a maximum context length of 1536 tokens for the bidirectional encoder and 768 tokens for the autoregressive decoder.
    • Shallow Encoder and Deep Decoder: Encoder depth is kept shallow while decoder depth is substantial (e.g., 5 encoder blocks / 30 decoder blocks for 1.1B parameters; 8 encoder blocks / 56 decoder blocks for 41.1B parameters), improving compute efficiency during training and decoding.
    • Multi-Query Attention: Standard multi-head attention maintains separate key and value heads for each query head. AlphaCode implements multi-query attention, sharing a single key head and a single value head across all query heads within each attention block (or 1 KV head per model parallel shard in 9B and 41B models). This reduces memory cache footprint and memory bandwidth bottlenecks during autoregressive decoding, increasing sampling throughput by over 12×12\times (from 0.37 samples/TPU-sec for standard multi-head attention to 4.74 samples/TPU-sec on the 1B model) with virtually no impact on solve rate (17.0% vs 17.3% at 10@10k10@10\text{k}).
    • Vocabulary: SentencePiece subword tokenizer with an 8,000-token vocabulary trained jointly on GitHub source code and CodeContests data.

  5. Knowl 5 — Fine-Tuning with Softmax Tempering and GOLD Off-Policy Reinforcement Learning

    model/method

    Competitive programming is a "one-of-many" task: a problem possesses many valid solution algorithms, and success requires finding only a single correct implementation within a submission budget nn. Standard maximum likelihood estimation (MLE) penalizes precision by forcing probability mass across all diverse training solutions (recall-oriented). To align the training objective with precision, AlphaCode applies a combination of Softmax Tempering and GOLD (Generation by Off-Policy Learning from Demonstrations):

    1. Softmax Tempering: During fine-tuning, logits are divided by a temperature T=0.2<1T = 0.2 < 1 before the softmax layer. This sharpens the training distribution, preventing the model from overfitting to diverse training solutions and smoothing the resulting inference distribution.
    2. GOLD Objective: An offline reinforcement learning policy-gradient objective applies an importance weight to the standard next-token log-likelihood gradients: ∇LGOLD(θ)=−∑s∈Solution tokensw(s)∇log⁡Pθ(s)\nabla \mathcal{L}_{\text{GOLD}}(\theta) = - \sum_{s \in \text{Solution tokens}} w(s) \nabla \log P_\theta(s) where w(s)=max⁡(Pθ(s)α,β)w(s) = \max(P_\theta(s)^\alpha, \beta) with α=0.5\alpha = 0.5 and clipping threshold β=0.05\beta = 0.05. The weight w(s)w(s) downweights or ignores tokens to which the model currently assigns low probability, allowing it to focus capacity on solution modes it can generate reliably.
    3. Tempering Warmup Transition: Because pre-trained models initially produce sharp logits under T=0.2T=0.2, applying GOLD immediately causes over-selection. A brief intermediate fine-tuning stage is used where softmax tempering is applied without GOLD before enabling full GOLD + tempering fine-tuning.

  6. Knowl 6 — Semantic Clustering of Candidate Programs via Learned Test Input Generation

    model/method

    Filtering candidate programs against public example tests eliminates ~99% of invalid samples, but leaves thousands of syntactically distinct candidates per problem. Selecting submissions purely at random from the filtered pool wastes the limited submission budget n≤10n \le 10 on semantically duplicate implementations. AlphaCode performs semantic clustering to select diverse candidates:

    1. Test-Input Generator Model: A separate encoder-decoder transformer (initialized from the GitHub pre-trained base model) is trained to generate input cases given problem descriptions, using problem example inputs, hidden platform inputs, and mutated inputs as training data.
    2. Execution and Output Signatures: The model generates synthetic test inputs (tuned to 50 inputs per problem). All filtered candidate programs (up to 8,192 samples) are executed on these 50 test inputs. Each program receives an execution signature consisting of its tuple of 50 outputs (or error codes).
    3. Equivalence Grouping and Cluster Ordering: Programs producing identical output signatures are grouped into the same cluster. Clusters are ordered by size in descending order.
    4. Cluster-Based Selection: Exactly one program is chosen from each cluster, proceeding from the largest cluster to the smallest. This heuristic exploits the property that correct algorithms tend to converge to identical correct behaviors (forming large clusters), whereas erroneous programs fail in divergent ways (forming small, fragmented clusters). If fewer than nn clusters exist, selection wraps around to the largest cluster, skipping already-selected samples.

  7. Knowl 7 — AlphaCode Competitive Performance on Codeforces Competitions

    empirical result

    AlphaCode was evaluated in a simulated live contest environment on all 10 Codeforces competitions held between 2021/12/01 and 2021/12/28 with more than 5,000 participants per contest. An ensemble of fine-tuned 41B and 9B models generated candidate solutions, filtered them on example tests, clustered on synthetic inputs, and submitted candidate programs under competition rules (including submission timing and incorrect submission penalties).

    Key results:

    • Percentile Ranking: Limiting to at most 10 submissions per problem, AlphaCode achieved an average ranking in the top 54.3% across the 10 contests (with individual contest rankings ranging from top 20.9% to top 73.9%).
    • Estimated Rating: The performance translates to an estimated Codeforces Elo rating of 1238, placing AlphaCode within the top 28% of active users who participated in at least one contest in the preceding six months.
    • Submission Efficiency: When limited to 10 attempts, AlphaCode used an average of 2.4 submissions per solved problem.
    • Unlimited Submissions: When allowed unlimited submissions with standard penalties, AlphaCode achieved an average ranking of top 48.8% (averaging 28.8 submissions per solved problem).

  8. Knowl 8 — Scaling Laws of Code Generation with Sample Budget and Compute

    empirical result

    Empirical evaluation on the CodeContests dataset demonstrates consistent scaling behaviors across sample budgets, model size, and compute budgets:

    • Log-Linear Sample Scaling: Both 10@k10@k (with filtering/clustering) and pass@kpass@k solve rates scale approximately log-linearly with the number of generated candidate samples kk across orders of magnitude (from k=1k=1 to k=106k=10^6).
    • Model Parameter Slopes: Larger models achieve both higher absolute solve rates and steeper scaling slopes in log-linear space. Consequently, a larger model reaches a given solve rate with exponentially fewer samples than a smaller model.
    • Compute Scaling: Solve rate scales approximately log-linearly with total training compute (TPU-days) and total sampling compute (TPU-seconds per problem). While larger models require more compute per sample, their higher sample efficiency makes larger models increasingly optimal as the total sampling compute budget grows.

  9. Knowl 9 — Performance and Component Ablations on CodeContests

    data/table

    The contribution of each architectural and training enhancement was evaluated via a cumulative build-up ablation on a 1.1B parameter model on the CodeContests validation set, reporting 10@k10@k solve rates across sample budgets k∈{1k,10k,100k,1M}k \in \{1\text{k}, 10\text{k}, 100\text{k}, 1\text{M}\} (with 95% confidence intervals from bootstrap subsampling):

    Fine-tuning setting 10@1K 10@10K 10@100K 10@1M
    No Enhancements 6.7% (6.5–6.8) 10.4% (9.6–11.0) 15.2% (14.3–15.9) 19.6% (18.2–20.4)
    + MLM 6.6% (6.2–7.0) 12.5% (12.1–12.7) 17.0% (16.5–17.2) 20.7% (19.1–21.3)
    + Tempering 7.7% (7.2–8.5) 13.3% (12.5–13.8) 18.7% (18.0–19.2) 21.9% (20.7–22.6)
    + Tags and Ratings 6.8% (6.4–7.0) 13.7% (12.8–14.9) 19.3% (18.1–20.0) 22.4% (21.3–23.0)
    + Value Conditioning 10.6% (9.8–11.1) 16.6% (16.4–16.9) 20.2% (19.6–20.7) 23.2% (21.7–23.9)
    + GOLD 12.4% (12.0–13.0) 17.3% (16.9–17.6) 21.5% (20.5–22.2) 24.2% (23.1–24.4)
    + Clustering 12.2% (10.8–13.4) 18.0% (17.3–18.8) 24.1% (23.2–25.0) 28.4% (27.5–29.3)

    At full scale, the 41B parameter model with clustering achieves a validation solve rate of 34.2% at 10@1M10@1\text{M} and 31.8% at 10@100k10@100\text{k}, and a test solve rate of 29.6% at 10@100k10@100\text{k}. Each enhancement contributes positively, with the combined improvements lifting the 1B model 10@100k10@100\text{k} solve rate from 15.2% to 24.1%.

  10. Knowl 10 — Non-Memorization Analysis via Longest Common Substrings

    empirical result

    To verify whether AlphaCode generates novel algorithmic solutions rather than retrieving memorized code from its training set, the lengths of Longest Common Substrings (LCS) between correct validation solutions and the training corpus (GitHub + CodeContests training set) were compared between model-generated solutions and human solutions:

    • Distribution Match: Model-generated solutions and human solutions share substrings with the training set at nearly identical distribution rates.
    • Tail Overlap: On CodeContests, less than 1.0% of model solutions shared a common substring longer than 600 characters with the training data, compared to approximately 3.0% of human solutions.
    • Qualitative Decomposition: Iterative LCS decomposition revealed that shared substrings consisted almost exclusively of syntactic boilerplate (e.g., standard fast I/O classes such as Python FastIO, input tokenizers, graph adjacency list setups), whereas core algorithmic logic, loops, and condition branching were novel combinations generated specifically for each problem.

  11. Knowl 11 — AlphaCode Sensitivity to Problem Descriptions and Metadata

    empirical result

    Ablation and perturbation experiments on problem descriptions demonstrate that AlphaCode genuinely reasons over description content rather than exploiting superficial dataset shortcuts:

    • Description Simplification: Replacing complex story narratives with concise algorithmic specifications (e.g., specifying "compute bitwise AND of all elements") improves sample solve rates dramatically (e.g., from 12.25% to 55.53% on Codeforces 1559A; from 0.95% to 85.38% on Nim).
    • Semantic Inversion: Rewording descriptions to require opposite or related logic drops sample solve rate to near zero (e.g., 17.1% on original down to 0.1% for opposite logic and 0.03% for underspecified descriptions on the Cherry problem).
    • Variable Renaming: Larger models are invariant to consistent variable renaming across descriptions, but experience sharp performance drops when variables are renamed inconsistently (rendering the problem ill-posed).
    • Section Removal: Removing problem components degrades 10@102410@1024 solve rate from 13.75% to 10.42% (without example I/O), 6.87% (without description), and 4.81% (without input/output specification).
    • Metadata Conditioning: Conditioning on randomly sampled tags per sample and uniformly sampled difficulty ratings (800–3500) outperforms conditioning on true problem tags (13.5% vs 13.3% at 10@102410@1024 on 1B), because randomizing metadata stimulates sampling diversity across algorithmic approaches.

  12. Knowl 12 — Validation Cross-Entropy Loss Divergence from Downstream Solve Rate

    limitation

    During fine-tuning on competitive programming corpora, validation language modeling cross-entropy loss is a poor proxy for downstream execution-based solve rate (10@k10@k or pass@kpass@k):

    • Observed Divergence: In fine-tuning runs, validation cross-entropy loss begins to increase after approximately 50,000 steps, which conventionally signals model overfitting. However, the execution solve rate on validation problems continues to increase steadily past 800,000 steps.
    • Underlying Mechanism: Competitive programming problems admit many distinct correct implementations in the training set. A model minimizing standard cross-entropy allocates probability mass across all observed human solutions (maximizing recall). As training progresses past the loss minimum, the model reallocates probability mass away from atypical solutions toward more canonical, typical solution modes. While this reallocation degrades the average likelihood on diverse human validation solutions, it substantially increases the probability that sampled programs will produce at least one correct implementation, thereby increasing n@kn@k solve rates.

Coverage note — Omitted comparative evaluations on external benchmarks (HumanEval and APPS) and problem-by-problem contest breakdown tables to prioritize the primary system architecture, algorithmic methods, training objectives, and scaling analyses.

References

  1. 1.Albert Ziegler. Research recitation: A first look at rote learning in GitHub Copilot suggestions. https://docs.github.com/en/github/copilot/research-recitation, 2021. Accessed: 2022-01-13.
  2. 2.M. Allamanis. The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, pages 143–153, 2019.
  3. 3.J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
  4. 4.G. A. Aye, S. Kim, and H. Li. Learning autocompletion from real-world datasets. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 131–139. IEEE, 2021.
  5. 5.M. Balog, A. L. Gaunt, M. Brockschmidt, S. Nowozin, and D. Tarlow. DeepCoder: Learning to write programs. arXiv preprint arXiv:1611.01989, 2016.
  6. 6.S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. van den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, D. de Las Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. Simonyan, J. W. Rae, E. Elsen, and L. Sifre. Improving language models by retrieving from trillions of tokens, 2021.
  7. 7.J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax.
  8. 8.T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
  9. 9.M. Bruch, M. Monperrus, and M. Mezini. Learning from examples to improve code completion systems. In Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering, pages 213–222, 2009.
  10. 10.E. Caballero, OpenAI, and I. Sutskever. Description2Code Dataset, 8 2016. URL https://github.com/ethancaballero/description2code.
  11. 11.N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650, 2021.
  12. 12.M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  13. 13.C. B. Clement, D. Drain, J. Timcheck, A. Svyatkovskiy, and N. Sundaresan. PyMT5: multi-mode translation of natural language and Python code with transformers. arXiv preprint arXiv:2010.03150, 2020.
  14. 14.K. Cobbe, V. Kosaraju, M. Bavarian, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  15. 15.R. Dabre and A. Fujita. Softmax tempering for training neural machine translation models. arXiv prepring arXiv:2009.09372, 2020.
  16. 16.J. Devlin, J. Uesato, S. Bhupatiraju, R. Singh, A.-r. Mohamed, and P. Kohli. RobustFill: Neural program learning under noisy I/O. In International conference on machine learning, pages 990–998. PMLR, 2017.
  17. 17.J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  18. 18.I. Drori and N. Verma. Solving linear algebra by program synthesis. arXiv preprint arXiv:2111.08171, 2021.
  19. 19.A. Ebtekar. How to interpret contest ratings. https://codeforces.com/blog/entry/68288, 2021. Accessed: 2021-12-04.
  20. 20.S. Edunov, M. Ott, M. Auli, and D. Grangier. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500, Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1045. URL https://aclanthology.org/D18-1045.
  21. 21.Facebook Hacker Cup. Facebook hacker cup. https://www.facebook.com/codingcompetitions/hacker-cup, 2021. Accessed: 2021-12-09.
  22. 22.A. Fan, M. Lewis, and Y. Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018.
  23. 23.Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, et al. CodeBERT: a pre-trained model for programming and natural languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1536–1547, 2020.
  24. 24.J. Ganitkevitch, B. Van Durme, and C. Callison-Burch. PPDB: The paraphrase database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 758–764, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL https://aclanthology.org/N13-1092.
  25. 25.D. Gershgorn. GitHub’s automatic coding tool rests on untested legal ground. https://www.theverge.com/2021/7/7/22561180/github-copilot-legal-copyright-fair-use-public-code, 2021. Accessed: 2022-01-10.
  26. 26.Google Code Jam. Google Code Jam. https://codingcompetitions.withgoogle.com/codejam, 2021. Accessed: 2021-12-09.
  27. 27.C. C. Green. Application of theorem proving to problem solving. In IJCAI, 1969.
  28. 28.S. Gulwani. Automating string processing in spreadsheets using input-output examples. ACM Sigplan Notices, 46(1):317–330, 2011.
  29. 29.S. Gulwani, O. Polozov, R. Singh, et al. Program synthesis. Foundations and Trends® in Programming Languages, 4(1-2):1–119, 2017.
  30. 30.D. Guo, A. Svyatkovskiy, J. Yin, N. Duan, M. Brockschmidt, and M. Allamanis. Learning to generate code sketches. arXiv preprint arXiv:2106.10158, 2021.
  31. 31.D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, et al. Measuring coding challenge competence with APPS. arXiv preprint arXiv:2105.09938, 2021.
  32. 32.T. Hennigan, T. Cai, T. Norman, and I. Babuschkin. Haiku: Sonnet for JAX, 2020. URL http://github.com/deepmind/dm-haiku.
  33. 33.A. Hindle, E. T. Barr, Z. Su, M. Gabel, and P. Devanbu. On the naturalness of software. In Proceedings of the 34th International Conference on Software Engineering, pages 837–847, 2012.
  34. 34.A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.
  35. 35.U. Hölzle. Meeting our match: Buying 100 percent renewable energy. https://www.blog.google/outreach-initiatives/environment/meeting-our-match-buying-100-percent-renewable-energy/, 2018. Accessed: 2022-01-10.
  36. 36.P.-S. Huang, R. Stanforth, J. Welbl, C. Dyer, D. Yogatama, S. Gowal, K. Dvijotham, and P. Kohli. Achieving verified robustness to symbol substitutions via interval bound propagation. In Empirical Methods in Natural Language Processing (EMNLP), pages 4081–4091, 2019.
  37. 37.ICPC. International collegiate programming contest. https://cse.umn.edu/cs/icpc, 2021. Accessed: 2021-12-04.
  38. 38.ICPC Factsheet. ICPC factsheet. https://icpc.global/worldfinals/pdf/Factsheet.pdf, 2020. Accessed: 2021-12-04.
  39. 39.ICPC Rules. ICPC rules. https://icpc.global/worldfinals/rules, 2021. Accessed: 2021-12-09.
  40. 40.IOI. International olympiad in informatics. https://ioinformatics.org/, 2021. Accessed: 2021-12-04.
  41. 41.N. P. Jouppi, D. H. Yoon, M. Ashcraft, M. Gottscho, T. B. Jablin, G. Kurian, J. Laudon, S. Li, P. Ma, X. Ma, et al. Ten lessons from three generations shaped Google’s TPUv4i. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), pages 1–14. IEEE, 2021.
  42. 42.J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
  43. 43.D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  44. 44.T. Kudo and J. Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018.
  45. 45.S. Kulal, P. Pasupat, K. Chandra, M. Lee, O. Padon, A. Aiken, and P. S. Liang. Spoc: Search-based pseudocode to code. Advances in Neural Information Processing Systems, 32, 2019.
  46. 46.W. Ling, P. Blunsom, E. Grefenstette, K. M. Hermann, T. Kočiský, F. Wang, and A. Senior. Latent predictor networks for code generation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 599–609, 2016. URL https://aclanthology.org/P16-1057.
  47. 47.I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
  48. 48.Z. Manna and R. J. Waldinger. Toward automatic program synthesis. Commun. ACM, 14(3):151–165, mar 1971. ISSN 0001-0782. doi: 10.1145/362566.362568. URL https://doi.org/10.1145/362566.362568.
  49. 49.N. D. Matsakis and F. S. Klock. The Rust language. ACM SIGAda Ada Letters, 34(3):103–104, 2014.
  50. 50.P. McKenzie. Falsehoods programmers believe about names. https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/, 2010. Accessed: 2022-01-10.
  51. 51.M. Mirzayanov. Codeforces: Results of 2020. https://codeforces.com/blog/entry/89502, 2020. Accessed: 2021-12-04.
  52. 52.V. Murali, L. Qi, S. Chaudhuri, and C. Jermaine. Neural sketch learning for conditional program generation. arXiv preprint arXiv:1703.05698, 2017.
  53. 53.Y. Nandwani, D. Jindal, Mausam, and P. Singla. Neural learning of one-of-many solutions for combinatorial problems in structured output spaces. In International Conference on Learning Representations, 2021.
  54. 54.R. Y. Pang and H. He. Text generation by learning from demonstrations. arXiv preprint arXiv:2009.07839, 2020.
  55. 55.H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri. An empirical cybersecurity evaluation of GitHub Copilot’s code contributions. CoRR, abs/2108.09293, 2021. URL https://arxiv.org/abs/2108.09293.
  56. 56.R. Puri, D. S. Kung, G. Janssen, W. Zhang, G. Domeniconi, V. Zolotov, J. Dolby, J. Chen, M. Choudhury, L. Decker, et al. Project CodeNet: A large-scale AI for code dataset for learning a diversity of coding tasks. arXiv preprint arXiv:2105.12655, 2021.
  57. 57.A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  58. 58.J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, et al. Scaling language models: Methods, analysis & insights from training Gopher. arXiv preprint arXiv:2112.11446, 2021.
  59. 59.V. Raychev, M. Vechev, and E. Yahav. Code completion with statistical language models. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 419–428, 2014.
  60. 60.S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma. CodeBLEU: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297, 2020.
  61. 61.M. Resnick, J. Maloney, A. Monroy-Hernández, N. Rusk, E. Eastmond, K. Brennan, A. Millner, E. Rosenbaum, J. Silver, B. Silverman, et al. Scratch: programming for all. Communications of the ACM, 52(11):60–67, 2009.
  62. 62.R. Robbes and M. Lanza. How program history can improve code completion. In 2008 23rd IEEE/ACM International Conference on Automated Software Engineering, pages 317–326. IEEE, 2008.
  63. 63.N. Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019.
  64. 64.A. Solar-Lezama. Program synthesis by sketching. University of California, Berkeley, 2008.
  65. 65.N. Sussman. Falsehoods programmers believe about time. https://infiniteundo.com/post/25326999628/falsehoods-programmers-believe-about-time, 2017. Accessed: 2022-01-10.
  66. 66.I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112, 2014.
  67. 67.A. Svyatkovskiy, S. K. Deng, S. Fu, and N. Sundaresan. IntelliCode compose: Code generation using transformer. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 1433–1443, 2020.
  68. 68.M. Tandy. Falsehoods programmers believe about addresses. https://www.mjt.me.uk/posts/falsehoods-programmers-believe-about-addresses/, 2013. Accessed: 2022-01-10.
  69. 69.L. Tang, E. Ke, N. Singh, N. Verma, and I. Drori. Solving probability and statistics problems by program synthesis. arXiv preprint arXiv:2111.08267, 2021.
  70. 70.D. Trivedi, J. Zhang, S.-H. Sun, and J. J. Lim. Learning to synthesize programs as interpretable and generalizable policies. In Advances in neural information processing systems, 2021.
  71. 71.A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.
  72. 72.O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019.
  73. 73.L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, Z. Kenton, S. Brown, W. Hawkins, T. Stepleton, C. Biles, A. Birhane, J. Haas, L. Rimell, L. A. Hendricks, W. S. Isaac, S. Legassick, G. Irving, and I. Gabriel. Ethical and social risks of harm from language models. CoRR, abs/2112.04359, 2021. URL https://arxiv.org/abs/2112.04359.
  74. 74.P. Yin and G. Neubig. A syntactic neural model for general-purpose code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 440–450, 2017.
  75. 75.M. Zavershynskyi, A. Skidanov, and I. Polosukhin. NAPS: Natural program synthesis dataset. arXiv preprint arXiv:1807.03168, 2018.

Citation

MLA
Li, Y., et al. “Competition-level Code Generation with AlphaCode”. Science, vol. 378, no. 6624, 2022, pp. 1092–97, https://doi.org/10.1126/science.abq1158.
APA
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., Hubert, T., Choy, P., de Masson d’Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., … Vinyals, O. (2022). Competition-level code generation with AlphaCode. Science, 378(6624), 1092–1097. https://doi.org/10.1126/science.abq1158
Chicago
Li, Y., D. Choi, J. Chung, et al. 2022. “Competition-level Code Generation with AlphaCode”. Science 378 (6624): 1092–97. https://doi.org/10.1126/science.abq1158.
Harvard
Li, Y. et al. (2022) “Competition-level code generation with AlphaCode”, Science, 378(6624), pp. 1092–1097. Available at: https://doi.org/10.1126/science.abq1158.
Vancouver
1. Li Y, Choi D, Chung J, et al (2022) Competition-level code generation with AlphaCode. Science 378:1092–1097

BibTeX

@article{Li_2022, title={Competition-level code generation with AlphaCode}, volume={378}, ISSN={1095-9203}, url={http://dx.doi.org/10.1126/science.abq1158}, DOI={10.1126/science.abq1158}, number={6624}, journal={Science}, publisher={American Association for the Advancement of Science (AAAS)}, author={Li, Yujia and Choi, David and Chung, Junyoung and Kushman, Nate and Schrittwieser, Julian and Leblond, Rémi and Eccles, Tom and Keeling, James and Gimeno, Felix and Dal Lago, Agustin and Hubert, Thomas and Choy, Peter and de Masson d’Autume, Cyprien and Babuschkin, Igor and Chen, Xinyun and Huang, Po-Sen and Welbl, Johannes and Gowal, Sven and Cherepanov, Alexey and Molloy, James and Mankowitz, Daniel J. and Sutherland Robson, Esme and Kohli, Pushmeet and de Freitas, Nando and Kavukcuoglu, Koray and Vinyals, Oriol}, year={2022}, month=Dec, pages={1092–1097} }
Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/