Back

Pre-trained models for natural language processing: A survey

Xipeng QiuTianxiang SunYige XuYunfan ShaoNing DaiXuanjing Huang

article2020Science China Technological Sciences1,742 citationsMost Influential Paper Award

Systematizes pre-trained language models using a structured four-perspective taxonomy while providing actionable techniques for downstream task adaptation and identifying open research challenges in natural language processing.

Listen

Building high-performing natural language processing systems historically required massive amounts of manually annotated data for each specific task. Because human annotation is expensive and slow, training deep neural networks from scratch often caused models to overfit on small datasets and generalize poorly in production. To resolve this bottleneck, the field has transitioned to using pre-trained models. These systems first learn general linguistic and factual representations from massive, unlabeled text datasets before being tailored to specific business applications.

The article systematically evaluates the development, architecture, adaptation strategies, and practical applications of pre-trained language models. Its objective is to provide a comprehensive structural taxonomy of existing approaches and outline key operational trade-offs and future development directions.

The article conducts an extensive analytical synthesis of recent literature, benchmark results, and system architectures across natural language processing. It examines models ranging from early static word embeddings to deep contextual architectures such as BERT, GPT, and T5, evaluating their training objectives, parameter scales, and benchmark performance across standard language understanding and generation evaluations.

The analysis yields five key findings. First, pre-trained contextual models deliver major performance gains over traditional training methods, with advanced models raising general language understanding benchmark scores from early baselines of roughly 72 to nearly 90 points. Second, self-supervised learning objectives—such as predicting masked words, permuted language modeling, and replaced token detection—enable effective training directly on raw, unannotated text corpora. Third, Transformer architectures have become the dominant standard due to their capacity to capture long-range dependencies, though their standard variants face computational limits on sequence lengths exceeding 512 tokens. Fourth, parameter-efficient adaptation methods (such as prompt tuning and task-specific adapters) achieve competitive accuracy with full model fine-tuning while modifying a tiny fraction of model weights, making large models viable at scale. Fifth, compression techniques—including knowledge distillation, quantization, and early-exit mechanisms—can speed up inference by roughly 1.6 to over 9 times while retaining the vast majority of baseline accuracy.

These findings indicate that adopting pre-trained foundations significantly lowers the development cost, timeline, and data requirements for deploying language technologies. However, the immense size of modern models introduces severe computational costs, operational latency, and deployment risks. Furthermore, deep contextual models remain vulnerable to subtle adversarial inputs and function largely as non-transparent systems, which creates operational and compliance risks in high-stakes environments.

Organizations should adopt pre-trained foundation models rather than building language systems from scratch. Teams should carefully match model selection to task requirements—using encoder architectures for extraction and classification, and sequence-to-sequence structures for text generation. When deploying to production or edge devices, organizations should implement compression techniques such as knowledge distillation and explore parameter-efficient tuning rather than full model re-training. Further research and engineering pilots should focus on improving model interpretability, developing robust adversarial defenses, and implementing non-quadratic architectures that efficiently process long-context documents.

Confidence in these performance findings is high across standard academic benchmarks, but decision-makers should exercise caution. Performance can degrade when models encounter domain-specific technical jargon, non-standard text distributions, or deliberate adversarial disruptions. Continued validation in live target environments remains essential.

Fudan Universityrepresentation learningtaxonomynatural language understandingpre-trained language modelsnatural language processingcontextual word embeddings
arXiv: 2003.08271
Cover for Pre-trained models for natural language processing: A survey

Abstract

Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy with four perspectives. Next, we describe how to adapt the knowledge of PTMs to the downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.

Topics

representation learningtaxonomynatural language understandingtext processing

Keywords

pre-trained language modelsnatural language processingcontextual word embeddingsUniversal Language Model Fine-tuningBERT modelspre-trained word embeddingsneural language modelscross-lingual natural language generationself-attention modelnatural language understanding tasks

Table of Contents

  • 1 Introduction
  • 2 Background
  • 2.1 Language Representation Learning
  • 2.2 Neural Contextual Encoders
  • 2.2.1 Sequence Models
  • 2.2.2 Non-Sequence Models
  • 2.2.3 Analysis
  • 2.3 Why Pre-training?
  • 2.4 A Brief History of PTMs for NLP
  • 2.4.1 First-Generation PTMs: Pre-trained Word Embeddings
  • 2.4.2 Second-Generation PTMs: Pre-trained Contextual Encoders
  • 3 Overview of PTMs
  • 3.1 Pre-training Tasks
  • 3.1.1 Language Modeling (LM)
  • 3.1.2 Masked Language Modeling (MLM)
  • 3.1.3 Permuted Language Modeling (PLM)
  • 3.1.4 Denoising Autoencoder (DAE)
  • 3.1.5 Contrastive Learning (CTL)
  • 3.1.6 Others
  • 3.2 Taxonomy of PTMs
  • 3.3 Model Analysis
  • 3.3.1 Non-Contextual Embeddings
  • 3.3.2 Contextual Embeddings
  • 4 Extensions of PTMs
  • 4.1 Knowledge-Enriched PTMs
  • 4.2 Multilingual and Language-Specific PTMs
  • 4.2.1 Multilingual PTMs
  • 4.2.2 Language-Specific PTMs
  • 4.3 Multi-Modal PTMs
  • 4.3.1 Video-Text PTMs
  • 4.3.2 Image-Text PTMs
  • 4.3.3 Audio-Text PTMs
  • 4.4 Domain-Specific and Task-Specific PTMs
  • 4.5 Model Compression
  • 4.5.1 Model Pruning
  • 4.5.2 Quantization
  • 4.5.3 Parameter Sharing
  • 4.5.4 Knowledge Distillation
  • 4.5.5 Module Replacing
  • 4.5.6 Early Exit
  • 5 Adapting PTMs to Downstream Tasks
  • 5.1 Transfer Learning
  • 5.2 How to Transfer?
  • 5.2.1 Choosing appropriate pre-training task, model architecture and corpus
  • 5.2.2 Choosing appropriate layers
  • 5.2.3 To tune or not to tune?
  • 5.3 Fine-Tuning Strategies
  • 5.3.1 Prompt-based Tuning
  • 6 Resources of PTMs
  • 7 Applications
  • 7.1 General Evaluation Benchmark
  • 7.2 Question Answering
  • 7.3 Sentiment Analysis
  • 7.4 Named Entity Recognition
  • 7.5 Machine Translation
  • 7.6 Summarization
  • 7.7 Adversarial Attacks and Defenses
  • 8 Future Directions
  • 9 Conclusion
  • References

Knowls

  1. Knowl 1 — Taxonomy of Pre-trained Models for Natural Language Processing

    definition

    Pre-trained models (PTMs) in natural language processing are categorized across four foundational dimensions and their downstream adaptation strategies:

    1. Representation Type:

      • Non-contextual representations: Static token embeddings mapped via a lookup table E∈RDe×∣V∣\mathbf{E} \in \mathbb{R}^{D_e \times |\mathcal{V}|}, invariant to context (e.g., Word2Vec, GloVe).
      • Contextual representations: Dynamic token vectors [h1,…,hT]=fenc(x1,…,xT)[ \mathbf{h}_1, \dots, \mathbf{h}_T] = f_{\text{enc}}(x_1, \dots, x_T) produced conditioned on the surrounding sequence context (e.g., ELMo, GPT, BERT).

    2. Backbone Architectures:

      • Recurrent/Sequence models: Multi-layer LSTMs or GRUs processing sequential tokens with locality bias.
      • Transformer Encoder: Bidirectional self-attention network modeling dependencies across all token pairs (e.g., BERT, RoBERTa).
      • Transformer Decoder: Left-to-right autoregressive attention using triangular causal masking to prevent lookahead (e.g., GPT series).
      • Transformer Encoder-Decoder (Full Transformer): Sequence-to-sequence structure comprising bidirectional encoder layers coupled to autoregressive decoder layers (e.g., MASS, BART, T5).

    3. Pre-training Task Types:

      • Supervised Learning: Supervised sequence objectives, such as neural machine translation (e.g., CoVe).
      • Unsupervised / Self-Supervised Learning: Tasks where supervisory signals are derived intrinsically from unlabeled text: autoregressive language modeling (LM), masked language modeling (MLM), translation language modeling (TLM), sequence-to-sequence MLM, permuted language modeling (PLM), denoising autoencoding (DAE), and contrastive learning (CTL) including replaced token detection (RTD), next sentence prediction (NSP), and sentence order prediction (SOP).

    4. Scenario Extensions:

      • Knowledge-Enriched: Integrating structured knowledge graph entities, triples, or semantic rules into representations.
      • Multilingual and Language-Specific: Cross-lingual understanding (XLU), cross-lingual generation (XLG), and monolingual models for specific languages.
      • Multi-Modal: Joint representation learning for video-text, image-text, and audio-text modalities.
      • Domain-Specific: Models pre-trained on specialized domain corpora (e.g., biomedical, scientific, clinical).
      • Model Compression & Early Exit: Methods to compress parameters or dynamically adapt layer depth at inference.

    5. Tuning Strategies:

      • Fine-Tuning: Full model tuning, multi-task tuning, two-stage transfer, or parameter-efficient adaptation modules (adapters/PALs).
      • Prompt-Tuning: Formatting downstream tasks into Cloze-style formulations via discrete prompts (manual, mined, or gradient-searched) or continuous virtual prompt embeddings.

  2. Knowl 2 — Mathematical Formulations of Pre-training Objectives

    equation

    For a sequence of tokens x=[x1,x2,…,xT]\mathbf{x} = [x_1, x_2, \dots, x_T], self-supervised pre-training tasks are formulated through the following loss objectives:

    1. Autoregressive Language Modeling (LM): LLM=−∑t=1Tlog⁡p(xt∣x<t)\mathcal{L}_{\text{LM}} = - \sum_{t=1}^T \log p(x_t \mid x_{<t}) where x<t=[x1,…,xt−1]x_{<t} = [x_1, \dots, x_{t-1}] and p(xt∣x<t)=gLM(fenc(x0:t−1))p(x_t \mid x_{<t}) = g_{\text{LM}}(f_{\text{enc}}(x_{0:t-1})).

    2. Masked Language Modeling (MLM): LMLM=−∑x^∈m(x)log⁡p(x^∣x∖m(x))\mathcal{L}_{\text{MLM}} = - \sum_{\hat{x} \in m(\mathbf{x})} \log p(\hat{x} \mid \mathbf{x} \setminus m(\mathbf{x})) where m(x)m(\mathbf{x}) denotes the subset of tokens selected for masking and x∖m(x)\mathbf{x} \setminus m(\mathbf{x}) denotes the unmasked context tokens.

    3. Sequence-to-Sequence Masked Language Modeling (Seq2Seq MLM): LS2SMLM=−∑t=ijlog⁡p(xt∣x∖xi:j,xi:t−1)\mathcal{L}_{\text{S2SMLM}} = - \sum_{t=i}^j \log p(x_t \mid \mathbf{x} \setminus \mathbf{x}_{i:j}, x_{i:t-1}) where xi:j\mathbf{x}_{i:j} is a contiguous masked nn-gram span from index ii to jj in x\mathbf{x}.

    4. Permuted Language Modeling (PLM): LPLM=−∑t=1Tlog⁡p(zt∣z<t)\mathcal{L}_{\text{PLM}} = - \sum_{t=1}^T \log p(z_t \mid z_{<t}) where z=perm(x)\mathbf{z} = \text{perm}(\mathbf{x}) is a random permutation order over token positions.

    5. Denoising Autoencoder (DAE): LDAE=−∑t=1Tlog⁡p(xt∣x^,x<t)\mathcal{L}_{\text{DAE}} = - \sum_{t=1}^T \log p(x_t \mid \hat{\mathbf{x}}, x_{<t}) where x^\hat{\mathbf{x}} is an input sequence corrupted via token masking, deletion, text infilling, sentence permutation, or document rotation.

    6. Contrastive Learning (CTL) & Deep InfoMax (DIM): LCTL=Ex,y+,y−[−log⁡exp⁡(s(x,y+))exp⁡(s(x,y+))+exp⁡(s(x,y−))]\mathcal{L}_{\text{CTL}} = \mathbb{E}_{\mathbf{x}, \mathbf{y}^+, \mathbf{y}^-} \left[ - \log \frac{\exp(s(\mathbf{x}, \mathbf{y}^+))}{\exp(s(\mathbf{x}, \mathbf{y}^+)) + \exp(s(\mathbf{x}, \mathbf{y}^-))} \right] LDIM=s(x^i:j,xi:j)−log⁡∑x~i:j∈Ns(x^i:j,x~i:j)\mathcal{L}_{\text{DIM}} = s(\hat{\mathbf{x}}_{i:j}, \mathbf{x}_{i:j}) - \log \sum_{\tilde{\mathbf{x}}_{i:j} \in \mathcal{N}} s(\hat{\mathbf{x}}_{i:j}, \tilde{\mathbf{x}}_{i:j}) where s(⋅,⋅)s(\cdot, \cdot) is a similarity score, y+\mathbf{y}^+ is a positive sample, y−\mathbf{y}^- is a negative sample, x^i:j\hat{\mathbf{x}}_{i:j} is context masked at span i:ji:j, xi:j\mathbf{x}_{i:j} is the true nn-gram span, and x~i:j∈N\tilde{\mathbf{x}}_{i:j} \in \mathcal{N} are sampled negative spans.

    7. Next Sentence Prediction (NSP) / Sentence Order Prediction (SOP): LNSP/SOP=−log⁡p(t∣x,y)\mathcal{L}_{\text{NSP/SOP}} = - \log p(t \mid \mathbf{x}, \mathbf{y}) where t=1t = 1 if segments x\mathbf{x} and y\mathbf{y} are continuous (for NSP) or in correct chronological order (for SOP), and t=0t = 0 otherwise.

    8. Replaced Token Detection (RTD): LRTD=−∑t=1Tlog⁡p(yt∣x^)\mathcal{L}_{\text{RTD}} = - \sum_{t=1}^T \log p(y_t \mid \hat{\mathbf{x}}) where yt=1(x^t=xt)y_t = \mathbf{1}(\hat{x}_t = x_t) indicates whether token x^t\hat{x}_t in the corrupted sequence x^\hat{\mathbf{x}} matches original token xtx_t.

  3. Knowl 3 — Benchmark Comparison of Representative Pre-trained Models

    data/table

    The table below compares representative pre-trained language models across architecture types, input formats, pre-training objectives, training corpora, parameter counts, GLUE benchmark average scores, and whether fine-tuning is standard practice:

    PTM Architecture Input Pre-Training Task Corpus Params GLUE FT?
    ELMo LSTM Text BiLM WikiText-103 – – No
    GPT Transformer Dec. Text LM BookCorpus 117M 72.8 Yes
    GPT-2 Transformer Dec. Text LM WebText 117M ∼\sim 1542M – No
    BERT Transformer Enc. Text MLM NSP WikiEn+BookCorpus 110M ∼\sim 340M 81.9* Yes
    InfoWord Transformer Enc. Text DIM+MLM WikiEn+BookCorpus =BERT 81.1* Yes
    RoBERTa Transformer Enc. Text MLM BookCorpus+CC-News+OpenWebText+STORIES 355M 88.5 Yes
    XLNet Two-Stream Trans. Enc. Text PLM WikiEn+BookCorpus+Giga5+ClueWeb+Common Crawl ≈\approxBERT 90.5§ Yes
    ELECTRA Transformer Enc. Text RTD+MLM Same as XLNet 335M 88.6 Yes
    UniLM Transformer Enc. Text MLM + NSP WikiEn+BookCorpus 340M 80.8 Yes
    MASS Transformer Text Seq2Seq MLM Task-dependent – – Yes
    BART Transformer Text DAE Same as RoBERTa 110% of BERT 88.4* Yes
    T5 Transformer Text Seq2Seq MLM C4 220M ∼\sim 11B 89.7* Yes
    ERNIE(THU) Transformer Enc. Text+Entities MLM+NSP+dEA WikiEn + Wikidata 114M 79.6 Yes
    KnowBERT Transformer Enc. Text MLM+NSP+EL WikiEn + WordNet/Wiki 253M ∼\sim 523M – Yes
    K-BERT Transformer Enc. Text+Triples MLM+NSP WikiZh+WebtextZh+CN-DBpedia+HowNet+MedKG =BERT – Yes
    KEPLER Transformer Enc. Text MLM+KE WikiEn + Wikidata/WordNet – – Yes
    WKLM Transformer Enc. Text MLM+ERD WikiEn + Wikidata =BERT – Yes
    CoLAKE Transformer Enc. Text+Triples MLM WikiEn + Wikidata =RoBERTa 86.3 Yes

    Note: GLUE score denotes average score on the GLUE benchmark (* indicates evaluation without the WNLI task; § indicates ensemble model score). 'FT?' indicates whether the model is intended primarily for fine-tuning.

  4. Knowl 4 — Model Compression Taxonomy for Pre-trained Language Models

    model/method

    Compression methods for reducing the memory footprint and latency of pre-trained language models are categorized into five distinct approaches:

    1. Model Pruning: Removing redundant parameters (individual weights, attention heads, neurons, or full transformer layers) either during pre-training or post-fine-tuning to reduce inference computational cost without altering the fundamental operator definitions.

    2. Weight Quantization: Converting floating-point model weights and activations from higher precision (e.g., 32-bit float) to lower precision integers (e.g., 8-bit or lower) via Quantization-Aware Training (QAT), Dynamic Quantization (DQ), or Hessian-Aware Quantization (HAWQ).

    3. Parameter Sharing: Reusing parameter matrices across layers (e.g., ALBERT's cross-layer parameter sharing) and decomposing high-dimensional embedding tables via factorized embedding parameterization. This drastically reduces the total parameter count, though floating-point operations (FLOPs) and inference latency remain similar.

    4. Knowledge Distillation (KD): Training a compact student network SS to reproduce the behavior of a large teacher model TT. Distillation paradigms comprise:

      • Soft target probability distillation: Optimizing cross-entropy loss between student predicted logits sis_i and teacher soft probabilities tit_i: LKD-CE=∑iti⋅log⁡(si)\mathcal{L}_{\text{KD-CE}} = \sum_i t_i \cdot \log(s_i)
      • Intermediate feature distillation: Minimizing mean squared error between teacher and student embedding outputs, hidden states, self-attention distributions, or self-attention value relations across layers (e.g., TinyBERT, MobileBERT, MiniLM).
      • Structural transfer: Distilling Transformer teachers into simpler recurrent (RNN) or convolutional (CNN) student architectures.

    5. Module Replacing: Progressively replacing large transformer blocks in a pre-trained network with compact substitute modules during training (e.g., BERT-of-Theseus), optimizing solely with downstream task loss.

  5. Knowl 5 — Quantitative Comparison of Compressed BERT Models

    data/table

    The table below compares compressed variants of BERT across compression paradigms, layer counts, optimization objectives, speedup factors, parameter footprints, source model, and GLUE benchmark scores:

    Method Type #Layers Loss Function Speed Up Params Source PTM GLUE
    BERT-Base Baseline 12 LMLM+LNSP\mathcal{L}_{\text{MLM}} + \mathcal{L}_{\text{NSP}} – 110M – 79.6
    BERT-Large Baseline 24 LMLM+LNSP\mathcal{L}_{\text{MLM}} + \mathcal{L}_{\text{NSP}} – 340M – 81.9
    Q-BERT Quantization 12 HAWQ + GWQ – BERT-Base BERT-Base ≈\approx99% BERT
    Q8BERT Quantization 12 DQ + QAT – BERT-Base BERT-Base ≈\approx99% BERT
    ALBERT Param. Sharing 12 LMLM+LSOP\mathcal{L}_{\text{MLM}} + \mathcal{L}_{\text{SOP}} ×5.6∼0.3\times 5.6 \sim 0.3 12 ∼\sim 235M – 89.4 (ens.)
    DistilBERT Distillation 6 LKD-CE+CosKD+LMLM\mathcal{L}_{\text{KD-CE}} + \text{CosKD} + \mathcal{L}_{\text{MLM}} ×1.63\times 1.63 66M BERT-Base 77.0 (dev)
    TinyBERT Distillation 4 MSEembed+MSEattn+MSEhidn+LKD-CE\text{MSE}_{\text{embed}} + \text{MSE}_{\text{attn}} + \text{MSE}_{\text{hidn}} + \mathcal{L}_{\text{KD-CE}} ×9.4\times 9.4 14.5M BERT-Base 76.5
    BERT-PKD Distillation 3 ∼\sim 6 LKD-CE+PTKD+LTask\mathcal{L}_{\text{KD-CE}} + \text{PTKD} + \mathcal{L}_{\text{Task}} ×3.73∼1.64\times 3.73 \sim 1.64 45.7 ∼\sim 67M BERT-Base 76.0 ∼\sim 80.6
    PD Distillation 6 LKD-CE+LTask+LMLM\mathcal{L}_{\text{KD-CE}} + \mathcal{L}_{\text{Task}} + \mathcal{L}_{\text{MLM}} ×2.0\times 2.0 67.5M BERT-Base 81.2
    MobileBERT Distillation 24 FMT+AT+PKT+LKD-CE+LMLM\text{FMT} + \text{AT} + \text{PKT} + \mathcal{L}_{\text{KD-CE}} + \mathcal{L}_{\text{MLM}} ×4.0\times 4.0 25.3M BERT-Large 79.7
    MiniLM Distillation 6 AT+AR\text{AT} + \text{AR} ×1.99\times 1.99 66M BERT-Base 81.0
    DualTrain Distillation 12 Dual Projection+LMLM\text{Dual Projection} + \mathcal{L}_{\text{MLM}} – 1.8 ∼\sim 19.2M BERT-Base 75.8 ∼\sim 81.9
    BERT-of-Theseus Module Replacing 6 LTask\mathcal{L}_{\text{Task}} ×1.94\times 1.94 66M BERT-Base 78.6

    Note: GLUE score is the average on 8 tasks (excluding WNLI). Speedup represents inference acceleration relative to the uncompressed base model. Parameter count illustrates structural footprint.

  6. Knowl 6 — Dynamic Early Exit Framework for Transformer Acceleration

    model/method

    Early exit accelerates Transformer inference by dynamically allowing an input sample to terminate computation at an intermediate layer ('off-ramp') rather than propagating through all LL layers.

    1. Off-Ramp Training Objective: Internal classifiers injected at intermediate layers are trained using a weighted sum of cross-entropy losses across MM off-ramps: Learly-exit=∑i=1Mwi⋅Li\mathcal{L}_{\text{early-exit}} = \sum_{i=1}^M w_i \cdot \mathcal{L}_i where wi≥0w_i \ge 0 is the weight for the ii-th off-ramp loss Li\mathcal{L}_i. Off-ramp training can be regularized via self-distillation against the final layer classifier, hidden state aggregation across past and future layers, or ensemble diversity objectives.

    2. Inference Exit Decision Strategies:

      • Entropy Thresholding: Exit if the Shannon entropy of the off-ramp's output distribution falls below a pre-set threshold τ\tau.
      • Max Softmax Probability: Exit if max⁡kpk≥τ\max_k p_k \ge \tau, where pkp_k is the class probability.
      • Patience-based Exiting: Exit if the predicted class label remains invariant across kk consecutive intermediate layers.
      • Ensemble Voting: Aggregate past off-ramp predictions to vote on an early exit trigger.
      • Window-based Uncertainty: Measure uncertainty across sliding token windows for sequence labeling early exit.

  7. Knowl 7 — Layer-Wise Representation Selection and Adaptation Schemes

    model/method

    When transferring an LL-layer pre-trained contextual encoder to downstream tasks, the target model g(⋅)g(\cdot) can ingest internal representations H(l)\mathbf{H}^{(l)} (1≤l≤L1 \le l \le L) through three structural configurations:

    1. Embedding Only: Using solely the non-contextual base embedding layer H(0)\mathbf{H}^{(0)}, training all downstream task layers from scratch.

    2. Top Layer: Feeding the final layer contextual hidden states H(L)\mathbf{H}^{(L)} directly into the task-specific prediction head g(H(L))g(\mathbf{H}^{(L)}).

    3. All-Layer Soft Combination (Scalar Mix): Extracting a soft combination across all layer hidden representations ht(l)\mathbf{h}_t^{(l)} for token position tt: rt=γ∑l=1Lαlht(l)\mathbf{r}_t = \gamma \sum_{l=1}^L \alpha_l \mathbf{h}_t^{(l)} where αl\alpha_l are softmax-normalized trainable scalar layer weights (∑l=1Lαl=1\sum_{l=1}^L \alpha_l = 1) and γ\gamma is a global learnable scaling parameter.

    Transfer Modes:

    • Feature Extraction (FE): Freezing all pre-trained parameters θPTM\theta_{\text{PTM}} and training only the parameters of g(⋅)g(\cdot), exposing internal representations as static features.
    • Fine-Tuning (FT): Unfreezing and jointly updating pre-trained weights along with task-specific parameters, which generally yields superior performance across NLP benchmarks.

  8. Knowl 8 — Categorization of Prompt-Based Tuning Methods

    model/method

    Prompt-based tuning reformulates downstream tasks into masked language modeling objectives by appending task-specific prompt templates to the input, bypassing or reducing parameter fine-tuning:

    1. Discrete Prompts: Explicit sequences of natural language tokens are prepended, interleaved, or appended to the input:

      • Manual Prompts: Handcrafted Cloze-style patterns designed per task (e.g., PET, LAMA, in-context learning in GPT-3).
      • Mined / Paraphrased Prompts: Prompts extracted from large corpora via text mining or expanded using paraphrasing models (e.g., LPAQA).
      • Gradient-Searched Prompts: Discrete token sequences searched automatically via gradient-guided token substitution (e.g., AutoPrompt).
      • LM-Generated Prompts: Generative pre-trained models (e.g., T5) deployed to generate optimal candidate prompt templates (e.g., LM-BFF).

    2. Continuous Prompts (Soft Prompts): Trainable continuous embedding vectors prepended to input sequence representations without requiring corresponding discrete vocabulary tokens:

      • Prefix-Tuning: Continuous task-specific prefix vectors prepended to the keys and values across Transformer layers for generation and summarization.
      • P-Tuning / WARP: Continuous virtual prompt embeddings inserted into input token embeddings, optimized with the core language model frozen.
      • Prompt Tuning Scale Equivalence: As pre-trained models scale to billions of parameters, parameter-efficient continuous prompt tuning matches the downstream performance of full-model fine-tuning.

  9. Knowl 9 — Taxonomy of Knowledge Injection Strategies for Pre-trained Models

    model/method

    To alleviate the lack of domain-specific and factual knowledge in general-purpose text representations, knowledge-enriched pre-trained models incorporate external Knowledge Bases (KBs) and Knowledge Graphs (KGs) via two paradigms:

    1. Pre-training Knowledge Injection:

      • Entity Linking & Embedding Fusion: Combining pre-trained knowledge graph entity embeddings with token representations aligned with entity mentions in text (e.g., ERNIE-THU, KnowBERT).
      • Joint Knowledge & Language Objectives: Jointly optimizing knowledge embedding loss (e.g., TransE objectives on triples) alongside masked language modeling loss (e.g., KEPLER).
      • Graph-Structured Inputs: Injecting knowledge graph triples directly into text sequences as structured trees or unified graph contexts, using specialized position matrices and visible attention masks to prevent semantic distortion (e.g., K-BERT, CoLAKE).
      • Entity Replacement Detection: Training models to distinguish authentic entity mentions from corrupt entity substitutions (e.g., WKLM).
      • Adapter-based Modular Injection: Training independent lightweight adapter modules for separate knowledge types while freezing PTM core weights to avoid catastrophic forgetting (e.g., K-Adapter).

    2. Post-Pre-training / Fine-Tuning Knowledge Infusion:

      • Injecting facts and commonsense knowledge (e.g., ConceptNet, ATOMIC) during downstream task fine-tuning or conditioning generation via knowledge graph language models (e.g., KGLM, LRLM).

  10. Knowl 10 — Open Challenges and Future Research Trajectories for Pre-trained Models

    limitation

    Five key structural challenges and future research directions for pre-trained language models include:

    1. Scaling and Computational Upper Bounds: PTM performance scales with model size (e.g., Megatron-LM, Turing-NLG) and training steps, but training demands immense compute. Efficient self-supervised sample-efficient objectives (e.g., ELECTRA) and optimized training skills (mixed precision, gradient accumulation, model parallelism) are required.

    2. Architectural Complexity of Transformers: Standard Transformer self-attention incurs O(T2)\mathcal{O}(T^2) time and memory complexity relative to sequence length TT, restricting context windows typically to 512 tokens. Developing sub-quadratic efficient attention mechanisms, non-Transformer architectures, and Neural Architecture Search (NAS) for PTMs is essential.

    3. Task Discrepancy & Compression Trade-offs: Architecture and distribution mismatches between general pre-training and downstream targets (e.g., bidirectional encoders vs. autoregressive generation) necessitate specialized pre-training designs and compressed models for edge and low-latency deployments.

    4. Knowledge Transfer Beyond Full Fine-Tuning: Full fine-tuning is parameter-inefficient as every downstream task maintains separate model copies. Modular parameter-efficient adaptation (adapters, projected attention layers, continuous prompt tuning) and knowledge distillation are critical alternatives.

    5. Interpretability, Reliability, and Robustness: Transformer decision-making is opaque, and reliance on attention weights for interpretability remains debated. PTMs are vulnerable to out-of-distribution shifts, misspellings, and adversarial attacks (such as universal adversarial triggers and BERT-Attack), requiring robust defense and adversarial pre-training.

Coverage note — Domain-specific empirical task overviews (e.g., specific downstream setups for sentiment analysis, machine translation, summarization, and named entity recognition) and specific third-party GitHub URL listings were omitted as non-foundational survey compendia.

References

  1. 1.Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. A convolutional neural network for modelling sentences. In ACL, 2014.
  2. 2.Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, pages 1746–1751, 2014.
  3. 3.Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In ICML, pages 1243–1252, 2017.
  4. 4.Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. Sequence to sequence learning with neural networks. In NeurIPS, pages 3104–3112, 2014.
  5. 5.Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Recurrent neural network for text classification with multi-task learning. In IJCAI, 2016.
  6. 6.Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pages 1631–1642. ACL, 2013.
  7. 7.Kai Sheng Tai, Richard Socher, and Christopher D. Manning. Improved semantic representations from tree-structured long short-term memory networks. In ACL, pages 1556–1566, 2015.
  8. 8.Diego Marcheggiani, Joost Bastings, and Ivan Titov. Exploiting semantics in neural machine translation with graph convolutional networks. In NAACL-HLT, pages 486–492, 2018.
  9. 9.Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2014.
  10. 10.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
  11. 11.Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In NeurIPS, 2013.
  12. 12.Jeffrey Pennington, Richard Socher, and Christopher D. Manning. GloVe: Global vectors for word representation. In EMNLP, 2014.
  13. 13.Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In NeurIPS, 2017.
  14. 14.Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL-HLT, 2018.
  15. 15.Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. URL https://s3-us-west-2.amazonaws.com/openai-assets/researchcovers/languageunsupervised/languageunderstandingpaper.pdf.
  16. 16.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
  17. 17.Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35 (8):1798–1828, 2013.
  18. 18.Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. Character-aware neural language models. In AAAI, 2016.
  19. 19.Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. TACL, 5:135–146, 2017. doi: https://doi.org/10.1162/tacl a 00051.
  20. 20.Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, 2016.
  21. 21.Sepp Hochreiter and Jurgen Schmidhuber. Long short-term ¨ memory. Neural Computation, 1997. doi: https://doi.org/10.1162/neco.1997.9.8.1735.
  22. 22.Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
  23. 23.Xiaodan Zhu, Parinaz Sobihani, and Hongyu Guo. Long short-term memory over recursive structures. In International Conference on Machine Learning, pages 1604–1612, 2015.
  24. 24.Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
  25. 25.Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. arXiv preprint arXiv:2106.04554, 2021.
  26. 26.Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. Star-transformer. In NAACL-HLT, pages 1315–1325, 2019.
  27. 27.Dumitru Erhan, Yoshua Bengio, Aaron C. Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? J. Mach. Learn. Res., 11:625–660, 2010. doi: https://dl.acm.org/doi/10.5555/1756006.1756025.
  28. 28.Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313 (5786):504–507, 2006. doi: https://doi.org/10.1126/science.1127647.
  29. 29.GE Hinton, JL McClelland, and DE Rumelhart. Distributed representations. In Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations, pages 77–109. 1986.
  30. 30.Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Chris- ´ tian Jauvin. A neural probabilistic language model. Journal of machine learning research, 3:1137–1155, 2003. doi: https://dl.acm.org/doi/10.5555/944919.944966.
  31. 31.Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, ´ Koray Kavukcuoglu, and Pavel P. Kuksa. Natural language processing (almost) from scratch. J. Mach. Learn. Res., 2011. doi: https://dl.acm.org/doi/10.5555/1953048.2078186.
  32. 32.Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents. In ICML, pages 1188–1196, 2014.
  33. 33.Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In NeurIPS, pages 3294–3302, 2015.
  34. 34.Oren Melamud, Jacob Goldberger, and Ido Dagan. Context2Vec: Learning generic context embedding with bidirectional LSTM. In CoNLL, pages 51–61, 2016.
  35. 35.Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In NeurIPS, pages 3079–3087, 2015.
  36. 36.Prajit Ramachandran, Peter J Liu, and Quoc Le. Unsupervised pretraining for sequence to sequence learning. In EMNLP, pages 383–391, 2017.
  37. 37.Alan Akbik, Duncan Blythe, and Roland Vollgraf. Contextual string embeddings for sequence labeling. In COLING, pages 1638–1649, 2018.
  38. 38.Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In ACL, pages 328–339, 2018.
  39. 39.Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. Cloze-driven pretraining of self-attention networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, EMNLP-IJCNLP, pages 5359–5368, 2019.
  40. 40.Wilson L. Taylor. “cloze procedure”: A new tool for measuring readability. Journalism Quarterly, 30(4):415–433, 1953. doi: https://doi.org/10.1177/107769905303000401.
  41. 41.Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MASS: masked sequence to sequence pre-training for language generation. In ICML, volume 97 of Proceedings of Machine Learning Research, pages 5926–5936, 2019.
  42. 42.Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.
  43. 43.Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  44. 44.Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. In NeurIPS, pages 13042–13054, 2019.
  45. 45.Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, et al. UniLMv2: Pseudo-masked language models for unified language model pre-training. arXiv preprint arXiv:2002.12804, 2020.
  46. 46.Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. In NeurIPS, pages 7057–7067, 2019.
  47. 47.Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77, 2019. doi: https://doi.org/10.1162/tacl a 00300.
  48. 48.Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Liwei Peng, and Luo Si. StructBERT: Incorporating language structures into pre-training for deep language understanding. In ICLR, 2020.
  49. 49.Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XLNet: Generalized autoregressive pretraining for language understanding. In NeurIPS, pages 5754–5764, 2019.
  50. 50.Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019.
  51. 51.Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In ICML, pages 5628–5637, 2019.
  52. 52.Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estimation. In NeurIPS, pages 2265–2273, 2013.
  53. 53.Michael Gutmann and Aapo Hyvarinen. Noise-contrastive ¨ estimation: A new estimation principle for unnormalized statistical models. In AISTATS, pages 297–304, 2010.
  54. 54.R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
  55. 55.Lingpeng Kong, Cyprien de Masson d’Autume, Lei Yu, Wang Ling, Zihang Dai, and Dani Yogatama. A mutual information maximization perspective of language representation learning. In ICLR, 2019.
  56. 56.Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In ICLR, 2020.
  57. 57.Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. In ICLR, 2020.
  58. 58.Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 2019.
  59. 59.Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  60. 60.Zewen Chi, Li Dong, Furu Wei, Wenhui Wang, Xian-Ling Mao, and Heyan Huang. Cross-lingual natural language generation via pre-training. In AAAI, 2019.
  61. 61.Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.08210, 2020.
  62. 62.Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard ´ Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116, 2019.
  63. 63.Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations, 2020.
  64. 64.Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to fine-tune BERT for text classification? In China National Conference on Chinese Computational Linguistics, pages 194–206, 2019.
  65. 65.Zhongyang Li, Xiao Ding, and Ting Liu. Story ending prediction by transferable bert. In IJCAI, pages 1800–1806, 2019.
  66. 66.Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, ´ Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don’t stop pretraining: Adapt language models to domains and tasks. In ACL, pages 8342–8360, 2020.
  67. 67.Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In ACL, 2019.
  68. 68.Asa Cooper Stickland and Iain Murray. BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning. In ICML, pages 5986–5995, 2019.
  69. 69.Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In ICML, pages 2790–2799, 2019.
  70. 70.Timo Schick and Hinrich Schutze. It’s not just size that mat- ¨ ters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 2339–2352. Association for Computational Linguistics, 2021.
  71. 71.Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 4222–4235. Association for Computational Linguistics, 2020.
  72. 72.Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723, 2020.
  73. 73.Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. WARP: word-level adversarial reprogramming. arXiv preprint arXiv:2101.00121, 2021.
  74. 74.Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021.
  75. 75.Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. arXiv preprint arXiv:2103.10385, 2021.
  76. 76.Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. ERNIE: enhanced language representation with informative entities. In ACL, 2019.
  77. 77.Matthew E. Peters, Mark Neumann, Robert L. Logan IV, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. Knowledge enhanced contextual word representations. In EMNLP-IJCNLP, 2019.
  78. 78.Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. K-BERT: Enabling language representation with knowledge graph. In AAAI, 2019.
  79. 79.Pei Ke, Haozhe Ji, Siyang Liu, Xiaoyan Zhu, and Minlie Huang. SentiLR: Linguistic knowledge enhanced language representation for sentiment analysis. arXiv preprint arXiv:1911.02493, 2019.
  80. 80.Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juanzi Li, and Jian Tang. KEPLER: A unified model for knowledge embedding and pre-trained language representation. arXiv preprint arXiv:1911.06136, 2019.
  81. 81.Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang. Colake: Contextualized language and knowledge embedding. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 3660–3670. International Committee on Computational Linguistics, 2020.
  82. 82.Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks. In EMNLP-IJCNLP, pages 2485–2494, 2019.
  83. 83.Julian Eisenschlos, Sebastian Ruder, Piotr Czapla, Marcin Kadras, Sylvain Gugger, and Jeremy Howard. MultiFiT: Efficient multi-lingual language model fine-tuning. In EMNLP-IJCNLP, pages 5701–5706, 2019.
  84. 84.Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. ERNIE: enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223, 2019.
  85. 85.Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. Pre-training with whole word masking for chinese BERT. arXiv preprint arXiv:1906.08101, 2019.
  86. 86.Junqiu Wei, Xiaozhe Ren, Xiaoguang Li, Wenyong Huang, Yi Liao, Yasheng Wang, Jiashu Lin, Xin Jiang, Xiao Chen, and Qun Liu. NEZHA: Neural contextualized representation for chinese language understanding. arXiv preprint arXiv:1909.00204, 2019.
  87. 87.Shizhe Diao, Jiaxin Bai, Yan Song, Tong Zhang, and Yonggang Wang. ZEN: pre-training chinese text encoder enhanced by n-gram representations. arXiv preprint arXiv:1911.00720, 2019.
  88. 88.Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. BERTje: A Dutch BERT model. arXiv preprint arXiv:1912.09582, 2019.
  89. 89.Louis Martin, Benjamin Muller, Pedro Javier Ortiz Su ¨ arez, ´ Yoann Dupont, Laurent Romary, Eric Villemonte de la Clerg- ´ erie, Djame Seddah, and Beno ´ ˆıt Sagot. CamemBERT: a tasty French language model. arXiv preprint arXiv:1911.03894, 2019.
  90. 90.Hang Le, Lo¨ıc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoˆıt Crabbe, Laurent Besacier, and Didier Schwab. FlauBERT: ´ Unsupervised language model pre-training for French. arXiv preprint arXiv:1912.05372, 2019.
  91. 91.Pieter Delobelle, Thomas Winters, and Bettina Berendt. RobBERT: a Dutch RoBERTa-based language model. arXiv preprint arXiv:2001.06286, 2020.
  92. 92.Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS, pages 13–23, 2019.
  93. 93.Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from transformers. In EMNLP-IJCNLP, pages 5099–5110, 2019.
  94. 94.Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. VisualBERT: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019.
  95. 95.Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter. Fusion of detected objects in text for visual question answering. In EMNLP-IJCNLP, pages 2131–2140, 2019.
  96. 96.Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VL-BERT: Pre-training of generic visual-linguistic representations. In ICLR, 2020.
  97. 97.Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. VideoBERT: A joint model for video and language representation learning. In ICCV, pages 7463–7472. IEEE, 2019.
  98. 98.Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Contrastive bidirectional transformer for temporal representation learning. arXiv preprint arXiv:1906.05743, 2019.
  99. 99.Yung-Sung Chuang, Chi-Liang Liu, and Hung-yi Lee. SpeechBERT: Cross-modal pre-trained language model for end-to-end spoken question answering. arXiv preprint arXiv:1910.11559, 2019.
  100. 100.Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 2019.
  101. 101.Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pre-trained language model for scientific text. In EMNLP-IJCNLP, pages 3613–3618, 2019.
  102. 102.Jieh-Sheng Lee and Jieh Hsiang. PatentBERT: Patent classification with fine-tuning a pre-trained BERT model. arXiv preprint arXiv:1906.02124, 2019.
  103. 103.Mitchell A Gordon, Kevin Duh, and Nicholas Andrews. Compressing BERT: Studying the effects of weight pruning on transfer learning. arXiv preprint arXiv:2002.08307, 2020.
  104. 104.Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-BERT: Hessian based ultra low precision quantization of BERT. In AAAI, 2020.
  105. 105.Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8BERT: Quantized 8bit BERT. arXiv preprint arXiv:1910.06188, 2019.
  106. 106.Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
  107. 107.Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for natural language understanding. arXiv preprint arXiv:1909.10351, 2019.
  108. 108.Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. arXiv preprint arXiv:2002.10957, 2020.
  109. 109.Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. BERT-of-Theseus: Compressing BERT by progressive module replacing. arXiv preprint arXiv:2002.02925, 2020.
  110. 110.Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. DeeBERT: Dynamic early exiting for accelerating BERT inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2246–2251, Online, July 2020. Association for Computational Linguistics.
  111. 111.Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A. Smith. The right tool for the job: Matching model and instance complexities. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 6640–6651. Association for Computational Linguistics, 2020.
  112. 112.Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, and Qi Ju. FastBERT: a self-distilling BERT with adaptive inference time. In ACL, 2020.
  113. 113.Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. Bert loses patience: Fast and robust inference with early exit. arXiv preprint arXiv:2006.04152, 2020.
  114. 114.Kaiyuan Liao, Yi Zhang, Xuancheng Ren, Qi Su, Xu Sun, and Bin He. A global past-future early exit method for accelerating inference of pre-trained language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 2013–2023. Association for Computational Linguistics, 2021.
  115. 115.Tianxiang Sun, Yunhua Zhou, Xiangyang Liu, Xinyu Zhang, Hao Jiang, Zhao Cao, Xuanjing Huang, and Xipeng Qiu. Early exiting with ensemble internal classifiers. arXiv preprint arXiv: 2105.13792, 2021.
  116. 116.Xiaonan Li, Yunfan Shao, Tianxiang Sun, Hang Yan, Xipeng Qiu, and Xuanjing Huang. Accelerating bert inference for sequence labeling via early-exit. In ACL, 2021.
  117. 117.Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In HLT-NAACL, pages 746–751, 2013.
  118. 118.Dana Rubinstein, Effi Levi, Roy Schwartz, and Ari Rappoport. How well do distributional models capture different types of semantic knowledge? In ACL, pages 726–730, 2015.
  119. 119.Abhijeet Gupta, Gemma Boleda, Marco Baroni, and Sebastian Pado. Distributional vectors encode referential attributes. In ´ EMNLP, pages 12–21, 2015.
  120. 120.Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations. In ICLR, 2019.
  121. 121.Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. In NAACL-HLT, pages 1073–1094, 2019.
  122. 122.Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Anna Korhonen, David R. Traum, and Llu´ıs Marquez, editors, ` ACL, pages 4593–4601, 2019.
  123. 123.Yoav Goldberg. Assessing BERT’s syntactic abilities. arXiv preprint arXiv:1901.05287, 2019.
  124. 124.Allyson Ettinger. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. TACL, 8:34–48, 2020.
  125. 125.John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. In NAACL-HLT, pages 4129–4138, 2019.
  126. 126.Ganesh Jawahar, Benoˆıt Sagot, and Djame Seddah. What does ´ BERT learn about the structure of language? In ACL, pages 3651–3657, 2019.
  127. 127.Taeuk Kim, Jihun Choi, Daniel Edmiston, and Sang goo Lee. Are pre-trained language models aware of phrases? simple but strong baselines for grammar induction. In ICLR, 2020.
  128. 128.Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and measuring the geometry of BERT. In NeurIPS, pages 8592–8600, 2019.
  129. 129.Fabio Petroni, Tim Rocktaschel, Sebastian Riedel, Patrick ¨ S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. Language models as knowledge bases? In EMNLP-IJCNLP, pages 2463–2473, 2019.
  130. 130.Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know? arXiv preprint arXiv:1911.12543, 2019.
  131. 131.Nina Porner, Ulli Waltinger, and Hinrich Sch ¨ utze. BERT is not ¨ a knowledge base (yet): Factual knowledge vs. name-based reasoning in unsupervised QA. CoRR, abs/1911.03681, 2019.
  132. 132.Nora Kassner and Hinrich Schutze. Negated LAMA: birds ¨ cannot fly. arXiv preprint arXiv:1911.03343, 2019.
  133. 133.Zied Bouraoui, Jose Camacho-Collados, and Steven Schock- ´ aert. Inducing relational knowledge from BERT. In AAAI, 2019.
  134. 134.Joe Davison, Joshua Feldman, and Alexander M. Rush. Commonsense knowledge mining from pretrained models. In EMNLP-IJCNLP, pages 1173–1178, 2019.
  135. 135.Anne Lauscher, Ivan Vulic, Edoardo Maria Ponti, Anna Korhonen, and Goran Glavas. Informing unsupervised pretraining with external linguistic knowledge. arXiv preprint arXiv:1909.02339, 2019.
  136. 136.Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv preprint arXiv:2002.01808, 2020.
  137. 137.Yoav Levine, Barak Lenz, Or Dagan, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. SenseBERT: Driving some sense into BERT. arXiv preprint arXiv:1908.05646, 2019.
  138. 138.Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, and Minlie Huang. A knowledge-enhanced pretraining model for commonsense story generation. arXiv preprint arXiv:2001.05139, 2020.
  139. 139.Bin He, Di Zhou, Jinghui Xiao, Xin Jiang, Qun Liu, Nicholas Jing Yuan, and Tong Xu. Integrating graph contextualized knowledge into pre-trained language models. arXiv preprint arXiv:1912.00147, 2019.
  140. 140.Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph and text jointly embedding. In EMNLP, pages 1591–1601, 2014.
  141. 141.Huaping Zhong, Jianwen Zhang, Zhen Wang, Hai Wan, and Zheng Chen. Aligning knowledge and text embeddings by entity descriptions. In EMNLP, pages 267–272, 2015.
  142. 142.Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, and Maosong Sun. Representation learning of knowledge graphs with entity descriptions. In IJCAI, 2016.
  143. 143.Jiacheng Xu, Xipeng Qiu, Kan Chen, and Xuanjing Huang. Knowledge graph representation with jointly structural and textual encoding. In IJCAI, pages 1318–1324, 2017.
  144. 144.An Yang, Quan Wang, Jing Liu, Kai Liu, Yajuan Lyu, Hua Wu, Qiaoqiao She, and Sujian Li. Enhancing pre-trained language representations with rich knowledge for machine reading comprehension. In ACL, pages 2346–2357, 2019.
  145. 145.Robert L. Logan IV, Nelson F. Liu, Matthew E. Peters, Matt Gardner, and Sameer Singh. Barack’s wife hillary: Using knowledge graphs for fact-aware language modeling. In ACL, 2019.
  146. 146.Hiroaki Hayashi, Zecong Hu, Chenyan Xiong, and Graham Neubig. Latent relation language models. In AAAI, 2019.
  147. 147.Manaal Faruqui and Chris Dyer. Improving vector space word representations using multilingual correlation. In EACL, pages 462–471, 2014.
  148. 148.Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Bilingual word representations with monolingual quality in mind. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 151–159, 2015.
  149. 149.Karan Singla, Dogan Can, and Shrikanth Narayanan. A multi- ˘ task approach to learning multilingual representations. In ACL, pages 214–220, 2018.
  150. 150.Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual BERT? In ACL, pages 4996–5001, 2019.
  151. 151.Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. Cross-lingual ability of multilingual BERT: An empirical study. In ICLR, 2020.
  152. 152.Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. Multilingual is not enough: BERT for Finnish. arXiv preprint arXiv:1912.07076, 2019.
  153. 153.Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE 2.0: A continual pretraining framework for language understanding. In AAAI, 2019.
  154. 154.Yuri Kuratov and Mikhail Arkhipov. Adaptation of deep bidirectional multilingual transformers for russian language. arXiv preprint arXiv:1905.07213, 2019.
  155. 155.Wissam Antoun, Fady Baly, and Hazem Hajj. AraBERT: Transformer-based model for Arabic language understanding. arXiv preprint arXiv:2003.00104, 2020.
  156. 156.Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Xilin Chen, and Ming Zhou. UniViLM: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.06353, 2020.
  157. 157.Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In AAAI, 2020.
  158. 158.Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: learning universal image-text representations. arXiv preprint arXiv:1909.11740, 2019.
  159. 159.Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. ClinicalBERT: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342, 2019.
  160. 160.Emily Alsentzer, John R. Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew B. A. McDermott. Publicly available clinical BERT embeddings. arXiv preprint arXiv:1904.03323, 2019.
  161. 161.Zongcheng Ji, Qiang Wei, and Hua Xu. BERT-based ranking for biomedical entity normalization. arXiv preprint arXiv:1908.03548, 2019.
  162. 162.Matthew Tang, Priyanka Gandhi, Md Ahsanul Kabir, Christopher Zou, Jordyn Blakey, and Xiao Luo. Progress notes classification and keyword extraction using attention-based deep learning models with BERT. arXiv preprint arXiv:1910.05786, 2019.
  163. 163.Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. arXiv preprint arXiv:1912.08777, 2019.
  164. 164.Shaolei Wang, Wanxiang Che, Qi Liu, Pengda Qin, Ting Liu, and William Yang Wang. Multi-task self-supervised learning for disfluency detection. In AAAI, 2019.
  165. 165.Cristian Bucilua, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In KDD, pages 535–541, 2006.
  166. 166.Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Deming Chen, Marianne Winslett, Hassan Sajjad, and Preslav Nakov. Compressing large-scale transformer-based models: A case study on BERT. arXiv preprint arXiv:2002.11985, 2020.
  167. 167.Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. In ICCV, pages 293–302, 2019.
  168. 168.Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  169. 169.Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for BERT model compression. In EMNLP-IJCNLP, pages 4323–4332, 2019.
  170. 170.Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: The impact of student initialization on knowledge distillation. arXiv preprint arXiv:1908.08962, 2019.
  171. 171.Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. MobileBERT: a compact task-agnostic BERT for resource-limited devices. arXiv preprint arXiv:2004.02984, 2020.
  172. 172.Sanqiang Zhao, Raghav Gupta, Yang Song, and Denny Zhou. Extreme language model compression with optimal subwords and shared projections. arXiv preprint arXiv:1909.11687, 2019.
  173. 173.Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in BERTology: What we know about how BERT works. arXiv preprint arXiv:2002.12327, 2020.
  174. 174.Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In NeurIPS, pages 14014–14024, 2019.
  175. 175.Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In ACL, pages 5797–5808, 2019.
  176. 176.Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In ICLR, 2019.
  177. 177.Wenhao Lu, Jian Jiao, and Ruofei Zhang. TwinBERT: Distilling knowledge to twin-structured BERT models for efficient retrieval. arXiv preprint arXiv:2002.06275, 2020.
  178. 178.Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, and Amelia Archer. Small and practical BERT models for sequence labeling. In EMNLP-IJCNLP, pages 3632–3636, 2019.
  179. 179.Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arXiv preprint arXiv:1904.09482, 2019.
  180. 180.Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task-specific knowledge from BERT into simple neural networks. arXiv preprint arXiv:1903.12136, 2019.
  181. 181.Yew Ken Chia, Sam Witteveen, and Martin Andrews. Transformer to CNN: Label-scarce distillation for efficient text classification. arXiv preprint arXiv:1909.03508, 2019.
  182. 182.Surat Teerapittayanon, Bradley McDanel, and H. T. Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 23rd International Conference on Pattern Recognition, ICPR 2016, Canc´un, Mexico, December 4-8, 2016, pages 2464–2469. IEEE, 2016.
  183. 183.Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. Shallow-deep networks: Understanding and mitigating network overthinking. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 3301–3310. PMLR, 2019.
  184. 184.Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983, 2016.
  185. 185.Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
  186. 186.Yijin Liu, Fandong Meng, Jie Zhou, Yufeng Chen, and Jinan Xu. Faster depth-adaptive transformers. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 13424–13432. AAAI Press, 2021.
  187. 187.Keli Xie, Siyuan Lu, Meiqi Wang, and Zhongfeng Wang. Elbert: Fast albert with confidence-window based early exit. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7713–7717. IEEE, 2021.
  188. 188.Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345–1359, 2009. doi: https://doi.org/10.1109/TKDE.2009.191.
  189. 189.Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. What do neural machine translation models learn about morphology? In ACL, pages 861–872, 2017.
  190. 190.Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. To tune or not to tune? adapting pretrained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP, RepL4NLP@ACL 2019, Florence, Italy, August 2, 2019, pages 7–14, 2019.
  191. 191.Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. Searching for effective neural extractive summarization: What works and what’s next. In ACL, pages 1049–1058, 2019.
  192. 192.Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tieyan Liu. Incorporating BERT into neural machine translation. In ICLR, 2020.
  193. 193.Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305, 2020.
  194. 194.Jason Phang, Thibault Fevry, and Samuel R Bowman. Sen- ´ tence encoders on STILTs: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088, 2018.
  195. 195.Siddhant Garg, Thuy Vu, and Alessandro Moschitti. Tanda: Transfer and adapt pre-trained transformer models for answer sentence selection. In AAAI, 2019.
  196. 196.Yige Xu, Xipeng Qiu, Ligao Zhou, and Xuanjing Huang. Improving BERT fine-tuning via self-ensemble and self-distillation. arXiv preprint arXiv:2002.10345, 2020.
  197. 197.Alexandra Chronopoulou, Christos Baziotis, and Alexandros Potamianos. An embarrassingly simple approach for transfer learning from pretrained language models. In NAACL-HLT, pages 2089–2095, 2019.
  198. 198.Xiang Lisa Li and Jason Eisner. Specializing word embeddings (for parsing) by information bottleneck. In EMNLP-IJCNLP, pages 2744–2754, 2019.
  199. 199.Teven Le Scao and Alexander M. Rush. How many data points is a prompt worth? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 2627–2636. Association for Computational Linguistics, 2021.
  200. 200.Timo Schick and Hinrich Schutze. Exploiting cloze-questions ¨ for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021, pages 255–269. Association for Computational Linguistics, 2021.
  201. 201.Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know. Trans. Assoc. Comput. Linguistics, 8:423–438, 2020.
  202. 202.Chi Sun, Luyao Huang, and Xipeng Qiu. Utilizing BERT for aspect-based sentiment analysis via constructing auxiliary sentence. In NAACL-HLT, 2019.
  203. 203.Guanghui Qin and Jason Eisner. Learning how to ask: Querying lms with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 5203–5212. Association for Computational Linguistics, 2021.
  204. 204.Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [MASK]: learning vs. learning to recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 5017–5033. Association for Computational Linguistics, 2021.
  205. 205.Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021.
  206. 206.Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. Allennlp: A deep semantic natural language processing platform. 2017.
  207. 207.Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. CTRL: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858, 2019.
  208. 208.Jesse Vig. A multiscale visualization of attention in the transformer model. In ACL, 2019.
  209. 209.Benjamin Hoover, Hendrik Strobelt, and Sebastian Gehrmann. exbert: A visual analysis tool to explore learned representations in transformers models. arXiv preprint arXiv:1910.05276, 2019.
  210. 210.Ziqing Yang, Yiming Cui, Zhipeng Chen, Wanxiang Che, Ting Liu, Shijin Wang, and Guoping Hu. Textbrewer: An open-source knowledge distillation toolkit for natural language processing. arXiv preprint arXiv:2002.12620, 2020.
  211. 211.Yuxuan Wang, Yutai Hou, Wanxiang Che, and Ting Liu. From static to dynamic word representations: a survey. International Journal of Machine Learning and Cybernetics, pages 1–20, 2020. doi: https://doi.org/10.1007/s13042-020-01069-8.
  212. 212.Qi Liu, Matt J Kusner, and Phil Blunsom. A survey on contextual embeddings. arXiv preprint arXiv:2003.07278, 2020.
  213. 213.Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
  214. 214.Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In NeurIPS, pages 3261–3275, 2019.
  215. 215.Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In Jian Su, Xavier Carreras, and Kevin Duh, editors, EMNLP, pages 2383–2392, 2016.
  216. 216.Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: A conversational question answering challenge. TACL, 7:249–266, 2019. doi: https://doi.org/10.1162/tacl a 00266.
  217. 217.Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP, pages 2369–2380, 2018.
  218. 218.Zhuosheng Zhang, Junjie Yang, and Hai Zhao. Retrospective reader for machine reading comprehension. arXiv preprint arXiv:2001.09694, 2020.
  219. 219.Ying Ju, Fubang Zhao, Shijie Chen, Bowen Zheng, Xuefeng Yang, and Yunfeng Liu. Technical report on conversational question answering. arXiv preprint arXiv:1909.10772, 2019.
  220. 220.Ming Tu, Kevin Huang, Guangtao Wang, Jing Huang, Xiaodong He, and Bowen Zhou. Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents. In AAAI, 2020.
  221. 221.Enkhbold Bataa and Joshua Wu. An investigation of transfer learning-based sentiment analysis in japanese. In ACL, 2019.
  222. 222.Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. BERT post-training for review reading comprehension and aspect-based sentiment analysis. In NAACL-HLT, 2019.
  223. 223.Alexander Rietzler, Sebastian Stabinger, Paul Opitz, and Stefan Engl. Adapt or get left behind: Domain adaptation through BERT language model finetuning for aspect-target sentiment classification. arXiv preprint arXiv:1908.11860, 2019.
  224. 224.Akbar Karimi, Leonardo Rossi, Andrea Prati, and Katharina Full. Adversarial training for aspect-based sentiment analysis with BERT. arXiv preprint arXiv:2001.11316, 2020.
  225. 225.Youwei Song, Jiahai Wang, Zhiwei Liang, Zhiyue Liu, and Tao Jiang. Utilizing BERT intermediate layers for aspect based sentiment analysis and natural language inference. arXiv preprint arXiv:2002.04815, 2020.
  226. 226.Xin Li, Lidong Bing, Wenxuan Zhang, and Wai Lam. Exploiting BERT for end-to-end aspect-based sentiment analysis. In W-NUT@EMNLP, 2019.
  227. 227.Xing Wu, Tao Zhang, Liangjun Zang, Jizhong Han, and Songlin Hu. ”mask and infill” : Applying masked language model to sentiment transfer. In IJCAI, 2019.
  228. 228.Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. Semi-supervised sequence tagging with bidirectional language models. In ACL, pages 1756–1765, 2017.
  229. 229.Liyuan Liu, Xiang Ren, Jingbo Shang, Xiaotao Gu, Jian Peng, and Jiawei Han. Efficient contextualized representation: Language model pruning for sequence labeling. In EMNLP, pages 1215–1225, 2018.
  230. 230.Kai Hakala and Sampo Pyysalo. Biomedical named entity recognition with multilingual BERT. In BioNLP Open Shared Tasks@EMNLP, pages 56–61, 2019.
  231. 231.Sergey Edunov, Alexei Baevski, and Michael Auli. Pre-trained language model representations for language generation. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, NAACL-HLT, pages 4052–4059, 2019.
  232. 232.Stephane Clinchant, Kweon Woo Jung, and Vassilina Nikoulina. On the use of BERT for neural machine translation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, Hong Kong, 2019.
  233. 233.Kenji Imamura and Eiichiro Sumita. Recycling a pre-trained BERT encoder for neural machine translation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, Hong Kong, November 2019.
  234. 234.Xingxing Zhang, Furu Wei, and Ming Zhou. HIBERT: Document level pre-training of hierarchical bidirectional transformers for document summarization. In ACL, pages 5059–5069, 2019.
  235. 235.Yang Liu and Mirella Lapata. Text summarization with pre-trained encoders. In EMNLP/IJCNLP, 2019.
  236. 236.Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuan-Jing Huang. Extractive summarization as text matching. In ACL, 2020.
  237. 237.Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is BERT really robust? natural language attack on text classification and entailment. In AAAI, 2019.
  238. 238.Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing NLP. In EMNLP-IJCNLP, pages 2153–2162, 2019.
  239. 239.Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai, Jia Li, Philip Yu, and Caiming Xiong. Adv-BERT: BERT is not robust on misspellings! generating nature adversarial samples on BERT. arXiv preprint arXiv:2003.04985, 2020.
  240. 240.Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. BERT-ATTACK: Adversarial attack against BERT using BERT. arXiv preprint arXiv:2004.09984, 2020.
  241. 241.Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. FreeLB: Enhanced adversarial training for natural language understanding. In ICLR, 2020.
  242. 242.Xiulei Liu, Hao Cheng, Peng cheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao. Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994, 2020.
  243. 243.Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019.
  244. 244.Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In ACL, pages 2978–2988, 2019.
  245. 245.Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
  246. 246.Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09282, 2017.
  247. 247.Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. Conditional BERT contextual augmentation. In International Conference on Computational Science, pages 84–95, 2019.
  248. 248.Varun Kumar, Ashutosh Choudhary, and Eunah Cho. Data augmentation using pre-trained transformer models. arXiv preprint arXiv:2003.02245, 2020.
  249. 249.Alejandro Barredo Arrieta, Natalia D´ıaz-Rodr´ıguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garc´ıa, Sergio Gil-Lopez, Daniel Molina, Richard ´ Benjamins, et al. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion, 58:82–115, 2020.
  250. 250.Sarthak Jain and Byron C Wallace. Attention is not explanation. In NAACL-HLT, pages 3543–3556, 2019.
  251. 251.Sofia Serrano and Noah A Smith. Is attention interpretable? In ACL, pages 2931–2951, 2019.

Citation

MLA
Qiu, X., et al. “Pre-trained Models for Natural Language Processing: A Survey”. Science China Technological Sciences, vol. 63, no. 10, 2020, pp. 1872–97, https://doi.org/10.1007/s11431-020-1647-3.
APA
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020). Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10), 1872–1897. https://doi.org/10.1007/s11431-020-1647-3
Chicago
Qiu, X., T. Sun, Y. Xu, Y. Shao, N. Dai, and X. Huang. 2020. “Pre-trained Models for Natural Language Processing: A Survey”. Science China Technological Sciences 63 (10): 1872–97. https://doi.org/10.1007/s11431-020-1647-3.
Harvard
Qiu, X. et al. (2020) “Pre-trained models for natural language processing: A survey”, Science China Technological Sciences, 63(10), pp. 1872–1897. Available at: https://doi.org/10.1007/s11431-020-1647-3.
Vancouver
1. Qiu X, Sun T, Xu Y, Shao Y, Dai N, Huang X (2020) Pre-trained models for natural language processing: A survey. Science China Technological Sciences 63:1872–1897

BibTeX

@article{Qiu_2020, title={Pre-trained models for natural language processing: A survey}, volume={63}, ISSN={1869-1900}, url={http://dx.doi.org/10.1007/s11431-020-1647-3}, DOI={10.1007/s11431-020-1647-3}, number={10}, journal={Science China Technological Sciences}, publisher={Springer Science and Business Media LLC}, author={Qiu, XiPeng and Sun, TianXiang and Xu, YiGe and Shao, YunFan and Dai, Ning and Huang, XuanJing}, year={2020}, month=Sept, pages={1872–1897} }
Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF