GLM: General Language Model Pretraining with Autoregressive Blank Infilling

Zhengxiao DuYujie QianXiao LiuMing DingJiezhong QiuZhilin YangJie Tang

article2021ACL1,989 citations

Introduces an autoregressive blank-infilling pretraining framework with 2D positional encodings that unifies natural language understanding, conditional generation, and unconditional generation while outperforming BERT, T5, and GPT across equivalent model scales.

Listen

Existing natural language processing models generally rely on distinct architectures tailored to specific task types, creating inefficiencies and operational silos. Masked language models excel at text understanding but cannot handle generation, left-to-right generative models struggle with bidirectional context understanding, and encoder-decoder systems often require substantial parameter overhead. The article addresses this fragmentation by evaluating whether a unified pretraining framework can achieve top-tier performance across natural language understanding, conditional sequence-to-sequence generation, and open-ended text generation within a single architecture.

To achieve this, the article introduces the General Language Model (GLM), which pretrains a standard transformer architecture using an autoregressive blank infilling objective. The approach randomly masks continuous text spans and trains the model to generate them sequentially while applying span shuffling and two-dimensional positional encodings to preserve context without exposing output lengths beforehand. The framework was evaluated across standard benchmarks including SuperGLUE, GLUE, abstractive summarization, question generation, and zero-shot language modeling, comparing directly against BERT, RoBERTa, T5, BART, and GPT under matched data and compute constraints.

Empirical evaluations demonstrate that GLM consistently outperforms conventional architectures across diverse applications. On the SuperGLUE understanding benchmark, GLM outperformed BERT by 4.6% to 5.0% given identical parameters and data, while matching or exceeding larger specialized architectures like RoBERTa, BART, and T5. In text generation tasks, multi-task GLM configurations matched or outperformed strong sequence-to-sequence baselines such as BART and T5 on abstractive summarization and question generation. Furthermore, GLM scaled to 410 million and 515 million parameters matched or surpassed GPT in zero-shot language modeling while retaining superior bidirectional context encoding capabilities.

These findings indicate that organizations can consolidate separate language understanding and text generation pipelines into a single model family, reducing training redundancy, lowering parameter overhead, and streamlining deployment infrastructure. The evidence strongly supports adopting autoregressive blank infilling and cloze-style task reformulation over standard classification layers, particularly for tasks involving multi-token predictions. Decision-makers planning model deployments should consider multi-task GLM variants as a versatile foundation, while maintaining awareness of task-specific trade-offs, such as document-level objectives slightly degrading conditional summarization precision.

Beijing Academy of Artificial Intelligence Massachusetts Institute of Technology Shanghai Qi Zhi Institute Tsinghua University encoder-decoder natural language understanding natural language generation General Language Model autoregressive blank infilling autoregressive pretraining

arXiv: 2103.10360 THUDM/GLM

Paper: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Jacob Devlin et al. (2019). Understanding BERT's masked autoencoding architecture and bidirectional pretraining is essential to grasp GLM's motivation to unify autoencoding with autoregressive blank infilling.
Paper: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, Colin Raffel et al. (2020). T5 establishes the span-corruption and encoder-decoder benchmark that GLM directly builds upon and modifies into a unified autoregressive blank-filling model.
Paper: BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension, Mike Lewis et al. (2020). BART introduces sequence-to-sequence denoising and span-infilling objectives that provide vital architectural and conceptual context for GLM.
Paper: XLNet: Generalized Autoregressive Pretraining for Language Understanding, Zhilin Yang et al. (2019). XLNet presents permutation autoregressive modeling and two-stream attention, motivating GLM's use of arbitrary span ordering and 2D positional encodings.
Paper: Improving Language Understanding by Generative Pre-Training, Alec Radford et al. (2018). Reading GPT-1 provides foundational knowledge of standard autoregressive language modeling, highlighting the generation paradigm GLM seeks to unify with bidirectional understanding.
Paper: SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems, Alex Wang et al. (2019). SuperGLUE serves as one of the primary evaluation benchmarks that GLM utilizes to prove its superiority over BERT and T5 on natural language understanding.
Paper: Transformer-XL: Attentive Language Models beyond a Fixed-Length Context, Zihang Dai et al. (2019). Transformer-XL introduces relative positional encodings and segment recurrence mechanisms crucial for understanding long-context handling in autoregressive transformers.

Paper: GLM-5: from Vibe Coding to Agentic Engineering, GLM-5-Team et al. (2026). GLM-5 represents the direct, scaled-up evolution of the General Language Model lineage toward mixture-of-experts architectures and autonomous agentic engineering.
Paper: CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation, Yue Wang et al. (2021). CodeT5 adapts unified pretraining paradigms with identifier-aware denoising to specialized software engineering tasks.
Paper: Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models, BIG-bench authors (2022). BIG-bench expands the evaluation of scaled generalist language models across hundreds of diverse reasoning and generation benchmarks.
Paper: Tulu 3: Pushing Frontiers in Open Language Model Post-Training, Nathan Lambert et al. (2024). Tulu 3 details modern post-training and alignment pipelines that take pretrained foundation models and adapt them for instruction following and verifiable reasoning.

Cover for GLM: General Language Model Pretraining with Autoregressive Blank Infilling

Abstract

There have been various types of pretraining architectures including autoencoding models (e.g., BERT), autoregressive models (e.g., GPT), and encoder-decoder models (e.g., T5). However, none of the pretraining frameworks performs the best for all tasks of three main categories including natural language understanding (NLU), unconditional generation, and conditional generation. We propose a General Language Model (GLM) based on autoregressive blank infilling to address this challenge. GLM improves blank filling pretraining by adding 2D positional encodings and allowing an arbitrary order to predict spans, which results in performance gains over BERT and T5 on NLU tasks. Meanwhile, GLM can be pretrained for different types of tasks by varying the number and lengths of blanks. On a wide range of tasks across NLU, conditional and unconditional generation, GLM outperforms BERT, T5, and GPT given the same model sizes and data, and achieves the best performance from a single pretrained model with 1.25x parameters of BERT Large , demonstrating its generalizability to different downstream tasks.

Topics

encoder-decoder natural language understanding natural language generation spoken language understanding n-gram language models statistical language models hierarchical model

Keywords

General Language Model autoregressive blank infilling autoregressive pretraining bidirectional attention conditional generation tasks multi-task pretraining SuperGLUE tasks long text generation encoder-decoder models

1 Introduction
2 GLM Pretraining Framework
2.1 Pretraining Objective
2.1.1 Autoregressive Blank Infilling
2.1.2 Multi-Task Pretraining
2.2 Model Architecture
2.2.1 2D Positional Encoding
2.3 Finetuning GLM
2.4 Discussion and Analysis
3 Experiments
3.1 Pretraining Setup
3.2 SuperGLUE
3.3 Multi-Task Pretraining
3.4 Ablation Study
4 Related Work
5 Conclusions
References
A Pretraining Setting
A.1 Datasets
A.2 Hyperparameters
A.3 Implementation
B Downstream Tasks
B.1 SuperGLUE
B.2 Sequence-to-Sequence
B.3 Text Infilling
B.4 Language Modeling
C Results on Other NLU Benchmarks
D Text Generation Samples

Knowls

Knowl 1 — Autoregressive Blank Infilling Pretraining Objective
model/method
General Language Model (GLM) trains a unified language model by blanking out continuous spans of tokens from an input sequence and autoregressively reconstructing them.

Given an input token sequence $\mathbf{x} = [x_1, \dots, x_n]$ , multiple text spans $\{s_1, \dots, s_m\}$ are sampled, where each span $s_i$ corresponds to a series of contiguous tokens $[s_{i,1}, \dots, s_{i, l_i}]$ in $\mathbf{x}$ . Each sampled span is replaced with a single special token $[\text{MASK}]$ , resulting in a corrupted text sequence $\mathbf{x}_{\text{corrupt}}$ . The span lengths are drawn from a Poisson distribution with mean $\lambda = 3$ , repeatedly sampling new spans until at least $15\%$ of original tokens are masked.

To capture dependencies between different masked spans, the order of spans is randomly permuted. Let $\mathcal{Z}_m$ denote the set of all possible permutations of the index sequence $[1, \dots, m]$ , and let $s_{z_{<i}} = [s_{z_1}, \dots, s_{z_{i-1}}]$ . The model parameters $\theta$ are optimized by maximizing the expected log-likelihood over all permutations:

$\max_{\theta} \mathbb{E}_{z \sim \mathcal{Z}_m} \left[ \sum_{i=1}^m \log p_\theta(s_{z_i} \mid \mathbf{x}_{\text{corrupt}}, s_{z_{<i}}) \right]$

Within each blank span $s_i = [s_{i,1}, \dots, s_{i, l_i}]$ , the tokens are generated strictly left-to-right according to the autoregressive factorization:

$p_\theta(s_i \mid \mathbf{x}_{\text{corrupt}}, s_{z_{<i}}) = \prod_{j=1}^{l_i} p(s_{i,j} \mid \mathbf{x}_{\text{corrupt}}, s_{z_{<i}}, s_{i,<j})$
Knowl 2 — 2D Positional Encoding in GLM
model/method
To handle variable-length masked spans without revealing the span length in advance during inference, GLM injects positional information using a 2D positional encoding scheme where each token is assigned two position IDs:
1. Position 1 (Inter-span position): Represents the position of the token in the corrupted text $\mathbf{x}_{\text{corrupt}}$ . For unmasked context tokens (Part A), this is their index in $\mathbf{x}_{\text{corrupt}}$ . For tokens in a masked span (Part B), this is the index of the corresponding $[\text{MASK}]$ token in $\mathbf{x}_{\text{corrupt}}$ .
2. Position 2 (Intra-span position): Represents the token's relative position within its span. For all tokens in Part A, Position 2 is set to $0$ . For tokens within each Part B span, Position 2 ranges from $1$ up to the span length $l_i$ .
Each position ID is mapped to an embedding vector via a separate learnable embedding table. Both positional vectors are added to the input token embedding vector. This ensures the model is not aware of the target span length prior to generation.
Knowl 3 — GLM Architecture and Masked Self-Attention
model/method
GLM employs a single Transformer architecture capable of bidirectional context encoding and unidirectional autoregressive decoding via structured self-attention masking:
- Input Partitioning: An input sequence $\mathbf{x}$ is divided into Part A (the corrupted text $\mathbf{x}_{\text{corrupt}}$ ) and Part B (the masked spans).
- Self-Attention Mask: Part A tokens attend bidirectionally to all tokens in Part A, but cannot attend to any tokens in Part B. Part B tokens can attend to all tokens in Part A, as well as to previously generated Part B tokens within previous spans and antecedent tokens within the current span. Part B tokens cannot attend to subsequent tokens in Part B.
- Span Boundary Tokens: Each span in Part B is prepended with a $[\text{START}]$ token for input and appended with an $[\text{END}]$ token as the generation target.
- Architectural Modifications: GLM uses Pre-Layer Normalization (reordering layer normalization and residual connections to prevent numerical errors in large models), a single linear layer for output token projection, and GeLU (Gaussian Error Linear Unit) activations.
Knowl 4 — Multi-Task Pretraining Objectives
model/method
To unify natural language understanding, conditional generation, and unconditional long text generation in a single model, GLM supports multi-task pretraining by combining the default short-span infilling objective with two longer-span objectives:
- Document-level Infilling: A single span whose length is sampled from a uniform distribution over $50\%\text{--}100\%$ of the original sequence length is masked. The model autoregressively generates the span given the remaining prefix, optimizing the model for long-text generation.
- Sentence-level Infilling: Masked spans are restricted to complete sentences. Multiple complete sentences are randomly masked until $15\%$ of the original tokens are covered, optimizing the model for sequence-to-sequence tasks whose outputs are full sentences or paragraphs.
During multi-task pretraining (yielding models such as $\text{GLM}_{\text{Doc}}$ and $\text{GLM}_{\text{Sent}}$ ), training batches sample short-span infilling and long-span objectives (document-level or sentence-level) with equal $50/50$ probability.
Knowl 5 — NLU Formulation as Cloze Blank Infilling with Verbalizers
model/method
GLM reformulates downstream Natural Language Understanding (NLU) classification tasks as autoregressive cloze blank infilling rather than training linear classifiers over sequence embeddings.

Given an input example $(\mathbf{x}, y)$ where $\mathbf{x}$ is the text and $y \in \mathcal{Y}$ is the true label:
1. The input $\mathbf{x}$ is converted into a natural language cloze prompt $c(\mathbf{x})$ containing a single $[\text{MASK}]$ token (e.g., for sentiment classification, "{SENTENCE}. It is really [MASK]").
2. Each candidate label $y \in \mathcal{Y}$ is mapped to a verbalizer answer $v(y)$ , which may consist of a single token (e.g., "good" or "bad") or multiple tokens.
3. The conditional probability of predicting label $y$ given $\mathbf{x}$ is:
$p(y \mid \mathbf{x}) = \frac{p(v(y) \mid c(\mathbf{x}))}{\sum_{y' \in \mathcal{Y}} p(v(y') \mid c(\mathbf{x}))}$

For single-token verbalizers, $p(v(y) \mid c(\mathbf{x}))$ is the logit of the verbalizer token. For multi-token verbalizers $[v(y)_1, \dots, v(y)_k]$ , $p(v(y) \mid c(\mathbf{x})) = \prod_{j=1}^k p(v(y)_j \mid c(\mathbf{x}), v(y)_{<j})$ , which GLM evaluates in a single forward pass. The model is finetuned by minimizing cross-entropy loss over $p(y \mid \mathbf{x})$ .

Knowl 6 — SuperGLUE Benchmark Performance

empirical result

GLM outperforms BERT, T5, BART, and RoBERTa on the SuperGLUE benchmark under identical pretraining data or comparable model sizes:

Model	ReCoRD	COPA	WSC	RTE	BoolQ	WiC	CB	MultiRC	Avg
	F1 / Acc.	Acc.	Acc.	Acc.	Acc.	Acc.	F1 / Acc.	F1a / EM
Pretrained on BookCorpus and Wikipedia
$\text{BERT}_{\text{Base}}$	65.4 / 64.9	66.0	65.4	70.0	74.9	68.8	70.9 / 76.8	68.4 / 21.5	66.1
$\text{GLM}_{\text{Base}}$	73.5 / 72.8	71.0	72.1	71.2	77.0	64.7	89.5 / 85.7	72.1 / 26.1	70.7
$\text{BERT}_{\text{Large}}$	76.3 / 75.6	69.0	64.4	73.6	80.1	71.0	94.8 / 92.9	71.9 / 24.1	72.0
$\text{UniLM}_{\text{Large}}$	80.0 / 79.1	72.0	65.4	76.5	80.5	69.7	91.0 / 91.1	77.2 / 38.2	74.1
$\text{GLM}_{\text{Large}}$	81.7 / 81.1	76.0	81.7	74.0	82.1	68.5	96.1 / 94.6	77.1 / 36.3	77.0
$\text{GLM}_{\text{Doc}}$	80.2 / 79.6	77.0	78.8	76.2	79.8	63.6	97.3 / 96.4	74.6 / 32.1	75.7
$\text{GLM}_{\text{Sent}}$	80.7 / 80.2	77.0	79.8	79.1	80.8	70.4	94.6 / 93.7	76.9 / 36.1	76.8
$\text{GLM}_{410\text{M}}$	81.5 / 80.9	80.0	81.7	79.4	81.9	69.0	93.2 / 96.4	76.2 / 35.5	78.0
$\text{GLM}_{515\text{M}}$	82.3 / 81.7	85.0	81.7	79.1	81.3	69.4	95.0 / 96.4	77.2 / 35.0	78.8
Pretrained on larger corpora ( $\approx 158\text{--}160\text{GB}$ )
$\text{T5}_{\text{Base}}$ (220M)	76.2 / 75.4	73.0	79.8	78.3	80.8	67.9	94.8 / 92.9	76.4 / 40.0	76.0
$\text{T5}_{\text{Large}}$ (770M)	85.7 / 85.0	78.0	84.6	84.8	84.3	71.6	96.4 / 98.2	80.9 / 46.6	81.2
$\text{BART}_{\text{Large}}$	88.3 / 87.8	60.0	65.4	84.5	84.3	69.0	90.5 / 92.9	81.8 / 48.0	76.0
$\text{RoBERTa}_{\text{Large}}$	89.0 / 88.4	90.0	63.5	87.0	86.1	72.6	96.1 / 94.6	84.4 / 52.9	81.5
$\text{GLM}_{\text{RoBERTa}}$	89.6 / 89.0	82.0	83.7	87.7	84.7	71.2	98.7 / 98.2	82.4 / 50.1	82.9

When pretrained on BookCorpus and Wikipedia, $\text{GLM}_{\text{Base}}$ exceeds $\text{BERT}_{\text{Base}}$ by $+4.6\%$ on average ( $70.7$ vs. $66.1$ ), and $\text{GLM}_{\text{Large}}$ exceeds $\text{BERT}_{\text{Large}}$ by $+5.0\%$ on average ( $77.0$ vs. $72.0$ ). Pretrained on 158GB of corpora for 250,000 steps, $\text{GLM}_{\text{RoBERTa}}$ achieves an average score of $82.9$ , outperforming $\text{RoBERTa}_{\text{Large}}$ ( $81.5$ ) and $\text{T5}_{\text{Large}}$ ( $81.2$ ).

Knowl 7 — Conditional Generation and Text Infilling Performance

empirical result

GLM demonstrates strong capabilities across sequence-to-sequence conditional generation and text infilling tasks:

Abstractive Summarization (CNN/DailyMail and XSum):

Model	CNN/DailyMail			XSum
	RG-1	RG-2	RG-L	RG-1	RG-2	RG-L
BERTSumAbs	41.7	19.4	38.8	38.8	16.3	31.2
UniLMv2_Base	43.2	20.4	40.1	44.0	21.1	36.1
T5_Large	42.5	20.7	39.8	40.9	17.3	33.0
BART_Large	44.2	21.3	40.9	45.1	22.3	37.3
GLM_RoBERTa	43.8	21.0	40.5	45.5	23.5	37.3

Summarization (Gigaword) and Question Generation (SQuAD 1.1) with BookCorpus/Wikipedia Pretraining:

Gigaword (ROUGE-1 / ROUGE-2 / ROUGE-L): $\text{GLM}_{\text{Large}}$ achieves $38.6 / 19.7 / 36.0$ , $\text{GLM}_{\text{Sent}}$ achieves $38.9 / 20.0 / 36.3$ , and $\text{GLM}_{410\text{M}}$ reaches $38.9 / 20.0 / 36.2$ , matching or exceeding MASS ( $37.7 / 18.5 / 34.9$ ) and UniLM ( $38.5 / 19.5 / 35.8$ ).
SQuAD Question Generation (BLEU-4 / METEOR / ROUGE-L): $\text{GLM}_{\text{Large}}$ achieves $22.4 / 25.2 / 50.4$ , $\text{GLM}_{\text{Sent}}$ achieves $22.6 / 25.4 / 50.4$ , and $\text{GLM}_{410\text{M}}$ achieves $22.9 / 25.6 / 50.5$ .

Yahoo Answers Text Infilling BLEU Scores:

Mask ratio	10%	20%	30%	40%	50%
BERT	82.8	66.3	50.3	37.4	26.2
BLM	86.5	73.2	59.6	46.8	34.8
GLM_Large	87.8	76.7	64.2	48.9	38.7
GLM_Doc	87.5	76.0	63.2	47.9	37.6

$\text{GLM}_{\text{Large}}$ outperforms Blank Language Model (BLM) across all masking ratios by $1.3$ to $3.9$ BLEU points.

Knowl 8 — Zero-Shot Language Modeling and Context Bidirectionality
empirical result
Zero-shot language modeling performance was evaluated using perplexity on a 20M-token held-out BookCorpus/Wikipedia test set and accuracy on the LAMBADA passage completion task:
- Standard $\text{GLM}_{\text{Large}}$ (pretrained without document-level generative loss) cannot perform zero-shot language modeling, yielding perplexity $> 100$ .
- At 340M parameters, $\text{GLM}_{\text{Doc}}$ achieves higher perplexity and lower LAMBADA accuracy than $\text{GPT}_{\text{Large}}$ when restricted to unidirectional generation, due to sharing model capacity between blank infilling and text generation.
- Scaling $\text{GLM}_{\text{Doc}}$ parameters to $410\text{M}$ ( $1.25\times \text{BERT}_{\text{Large}}$ ) approaches $\text{GPT}_{\text{Large}}$ performance, while $\text{GLM}_{515\text{M}}$ ( $1.5\times \text{BERT}_{\text{Large}}$ ) outperforms $\text{GPT}_{\text{Large}}$ on LAMBADA accuracy (approx. $54.5\%$ vs. $51.5\%$ ) and perplexity.
- Bidirectional Attention over Context: Encoding the prompt context (Part A) bidirectionally rather than unidirectionally improves language modeling perplexity and accuracy, allowing $\text{GLM}_{410\text{M}}$ with bidirectional context encoding to outperform unidirectional $\text{GPT}_{\text{Large}}$ .
- 2D Positional Encoding Impact: Removing 2D positional encoding from $\text{GLM}_{\text{Doc}}$ increases unidirectional test perplexity from $\approx 15.1$ to $\approx 15.3$ and lowers LAMBADA accuracy from $\approx 46.5\%$ to $\approx 45.0\%$ .

Knowl 9 — Ablation Analysis of GLM Components

empirical result

An ablation study on the SuperGLUE development set quantifies the contributions of GLM's design elements:

Model	ReCoRD	COPA	WSC	RTE	BoolQ	WiC	CB	MultiRC	Avg
	F1 / Acc.	Acc.	Acc.	Acc.	Acc.	Acc.	F1 / Acc.	F1a / EM
$\text{BERT}_{\text{Large}}$	76.3 / 75.6	69.0	64.4	73.6	80.1	71.0	94.8 / 92.9	71.9 / 24.1	72.0
$\text{BERT}_{\text{Large}}$ (reproduced)	82.1 / 81.5	63.0	63.5	72.2	80.8	68.7	80.9 / 85.7	77.0 / 35.2	71.2
$\text{BERT}_{\text{Large}}$ (cloze)	70.0 / 69.4	80.0	76.0	72.6	78.1	70.5	93.5 / 91.1	70.0 / 23.1	73.2
$\text{GLM}_{\text{Large}}$	81.7 / 81.1	76.0	81.7	74.0	82.1	68.5	96.1 / 94.6	77.1 / 36.3	77.0
– cloze finetune	81.3 / 80.6	62.0	63.5	66.8	80.5	65.0	89.2 / 91.1	72.3 / 27.9	70.0
– shuffle spans	82.0 / 81.4	61.0	79.8	54.5	65.8	56.3	90.5 / 92.9	76.7 / 37.6	68.5
+ sentinel tokens	81.8 / 81.3	69.0	78.8	77.3	81.2	68.0	93.7 / 94.6	77.5 / 37.7	76.0

Key observations:

Span Shuffling: Removing random span permutation (generating masked spans strictly in left-to-right document order) leads to an $8.5$ -point average performance drop on SuperGLUE ( $77.0 \to 68.5$ ).
Cloze Formulation: Finetuning GLM using a sequence classifier over $[\text{CLS}]$ representations rather than autoregressive cloze blank infilling drops performance by $7.0$ points ( $77.0 \to 70.0$ ).
Shared [MASK] vs. Distinct Sentinel Tokens: Replacing a single $[\text{MASK}]$ token with multiple unique sentinel tokens (as in T5) decreases average performance ( $77.0 \to 76.0$ ), as learning multiple sentinel embeddings wastes capacity on tokens unused in single-blank downstream tasks.
Multi-Token Answers: Cloze-style BERT degrades on multi-token verbalizers (ReCoRD F1 drops from $82.1$ to $70.0$ ), whereas GLM handles multi-token targets autoregressively without performance loss.

Knowl 10 — Downstream Performance on GLUE and SQuAD Benchmarks

empirical result

When evaluated on the GLUE benchmark and the SQuAD extractive question answering benchmark, GLM consistently outperforms BERT under identical model architectures and parameter counts:

GLUE Benchmark Development Set Results:

Model	MNLI	QNLI	QQP	RTE	SST-2	MRPC	CoLA	STS-B	Avg
$\text{BERT}_{\text{Large}}$	86.6	92.3	91.3	73.6	93.2	88.0	60.6	90.0	84.4
$\text{GLM}_{\text{Large}}$	86.7	92.8	91.5	74.0	93.5	90.0	61.4	90.7	85.1

SQuAD v1.1 and v2.0 Development Sets (Exact Match / F1):

Model	SQuAD v1.1 (EM / F1)	SQuAD v2.0 (EM / F1)
$\text{BERT}_{\text{Base}}$	80.8 / 88.5	73.7 / 76.3
$\text{GLM}_{\text{Base}}$	81.5 / 88.6	74.7 / 77.8
$\text{BERT}_{\text{Large}}$	84.1 / 90.9	79.0 / 81.8
$\text{GLM}_{\text{Large}}$	85.4 / 91.6	80.3 / 83.3

On GLUE, $\text{GLM}_{\text{Large}}$ achieves an average score of $85.1$ compared to $84.4$ for $\text{BERT}_{\text{Large}}$ . On SQuAD v2.0, $\text{GLM}_{\text{Large}}$ improves Exact Match / F1 by $+1.3 / +1.5$ over $\text{BERT}_{\text{Large}}$ .

Coverage note — Qualitative text generation samples from Appendix D were omitted as they provide illustrative outputs rather than reusable quantitative or algorithmic contributions.

References

1.Ben Athiwaratkun, Cicero dos Santos, Jason Krone, and Bing Xiang. 2020. Augmented natural language for generative sequence labeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 375–385.
2.Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, and Hsiao-Wuen Hon. 2020. Unilmv2: Pseudo-masked language models for unified language model pre-training. In ICML 2020, volume 119, pages 642–652.
3.Bin Bi, Chenliang Li, Chen Wu, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. 2020. PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation. In EMNLP 2020, pages 8681–8691.
4.Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In NeurIPS 2020.
5.Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14.
6.Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. In ICLR 2020.
7.Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177–190. Springer.
8.Michael Denkowski and Alon Lavie. 2014. Meteor Universal: Language Specific Translation Evaluation for Any Target Language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376–380.
9.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL 2019, pages 4171–4186.
10.Chris Donahue, Mina Lee, and Percy Liang. 2020. Enabling language models to fill in the blanks. pages 2492–2501.
11.Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In NeurIPS 2019, pages 13042–13054.
12.Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to Ask: Neural Question Generation for Reading Comprehension. In ACL 2017, pages 1342–1352.
13.Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus.
14.Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. ArXiv, abs/2006.03654.
15.Dan Hendrycks and Kevin Gimpel. 2016. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415.
16.Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving Pre-training by Representing and Predicting Spans. Trans. Assoc. Comput. Linguistics, 8:64–77.
17.Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In ICLR 2020.
18.Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In ACL 2020, pages 7871–7880.
19.Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. pages 74–81.
20.Yang Liu and Mirella Lapata. 2019. Text Summarization with Pretrained Encoders. In EMNLP 2019, pages 3730–3740.
21.Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
22.Joel Mackenzie, Rodger Benham, Matthias Petri, Johanne R. Trippas, J. Shane Culpepper, and Alistair Moffat. 2020. CC-News-En: A Large English News Corpus. In CIKM 2020, pages 3077–3084.
23.Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. In EMNLP 2018, pages 1797–1807.
24.Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cícero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2020. Structured Prediction as Translation between Augmented Natural Languages.
25.Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In ACL 2016.
26.Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A Method for Automatic Evaluation of Machine Translation. In ACL 2002, pages 311–318.
27.Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings.
28.Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018a. Improving Language Understanding by Generative Pre-Training.
29.Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018b. Language models are unsupervised multitask learners.
30.Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res., 21:140:1–140:67.
31.Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know What You Don’t Know: Unanswerable Questions for SQuAD. In ACL 2018, pages 784–789.
32.Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP 2016, pages 2383–2392.
33.Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In KDD 2020, pages 3505–3506.
34.Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In EMNLP 2015, pages 379–389.
35.Timo Schick and Hinrich Schütze. 2020a. Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference. pages 255–269.
36.Timo Schick and Hinrich Schütze. 2020b. It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners. pages 2339–2352.
37.Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. In ACL 2017, pages 1073–1083.
38.Tianxiao Shen, Victor Quach, Regina Barzilay, and Tommi S. Jaakkola. 2020. Blank language models. pages 5186–5198.
39.Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. CoRR, abs/1909.08053.
40.Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In EMNLP 2013, pages 1631–1642.
41.Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. MASS: Masked Sequence to Sequence Pre-training for Language Generation. In ICML 2019, volume 97, pages 5926–5936.
42.Trieu H. Trinh and Quoc V. Le. 2019. A Simple Method for Commonsense Reasoning. arXiv:1806.02847 [cs].
43.Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. In NeurIPS 2019, pages 3261–3275.
44.Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In ICLR 2019, pages 353–355.
45.Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In NAACL 2018, pages 1112–1122.
46.Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In NeurIPS 2019, pages 5754–5764.
47.Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated convolutions. In ICML 2017, volume 70, pages 3881–3890.
48.Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020. PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization. In ICML 2020, pages 11328–11339.
49.Wanrong Zhu, Zhiting Hu, and Eric Xing. 2019. Text infilling. arXiv preprint arXiv:1901.00158.
50.Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV 2015, pages 19–27.

Citation

MLA

Du, Z., et al. “GLM: General Language Model Pretraining with Autoregressive Blank Infilling”. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 320–35, https://doi.org/10.18653/v1/2022.acl-long.26.

APA

Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022). GLM: General Language Model Pretraining with Autoregressive Blank Infilling. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 320–335. https://doi.org/10.18653/v1/2022.acl-long.26

Chicago

Du, Z., Y. Qian, X. Liu, et al. 2022. “GLM: General Language Model Pretraining with Autoregressive Blank Infilling”. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 320–35. https://doi.org/10.18653/v1/2022.acl-long.26.

Harvard

Du, Z. et al. (2022) “GLM: General Language Model Pretraining with Autoregressive Blank Infilling”, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pp. 320–335. Available at: https://doi.org/10.18653/v1/2022.acl-long.26.

Vancouver

1. Du Z, Qian Y, Liu X, Ding M, Qiu J, Yang Z, Tang J (2022) GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pp 320–335

BibTeX

@inproceedings{du-etal-2022-glm,
    title = "{GLM}: General Language Model Pretraining with Autoregressive Blank Infilling",
    author = "Du, Zhengxiao  and
      Qian, Yujie  and
      Liu, Xiao  and
      Ding, Ming  and
      Qiu, Jiezhong  and
      Yang, Zhilin  and
      Tang, Jie",
    editor = "Muresan, Smaranda  and
      Nakov, Preslav  and
      Villavicencio, Aline",
    booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = may,
    year = "2022",
    address = "Dublin, Ireland",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.acl-long.26/",
    doi = "10.18653/v1/2022.acl-long.26",
    pages = "320--335"
}

Metadata:ACL Anthology

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

GLM: General Language Model Pretraining with Autoregressive Blank Infilling

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Autoregressive Blank Infilling Pretraining Objective

Knowl 2 — 2D Positional Encoding in GLM

Knowl 3 — GLM Architecture and Masked Self-Attention

Knowl 4 — Multi-Task Pretraining Objectives

Knowl 5 — NLU Formulation as Cloze Blank Infilling with Verbalizers

Knowl 6 — SuperGLUE Benchmark Performance

Knowl 7 — Conditional Generation and Text Infilling Performance

Knowl 8 — Zero-Shot Language Modeling and Context Bidirectionality

Knowl 9 — Ablation Analysis of GLM Components

Knowl 10 — Downstream Performance on GLUE and SQuAD Benchmarks

References

Citation

BibTeX

Source Code

Access the Paper