Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference cover

📄 Executive Summary

Applying natural language processing to real-world tasks often faces a severe bottleneck: annotating large volumes of high-quality data is expensive, time-consuming, and impractical across numerous languages and domains. While standard supervised learning performs poorly with only a handful of examples, pretrained language models possess substantial latent knowledge that typically goes underutilized unless prompted effectively.

This article set out to evaluate whether reformulating classification inputs into natural language cloze questions—termed Pattern-Exploiting Training (PET)—can combine the benefits of task descriptions with supervised fine-tuning to outperform conventional learning methods in low-resource settings.

The authors conducted an experimental evaluation comparing PET and an iterative extension (iPET) against standard supervised baselines, unsupervised prompting, and leading semi-supervised approaches. The evaluation covered four English datasets spanning sentiment analysis, news classification, question categorization, and natural language inference, as well as a multilingual stance detection dataset covering German, French, and Italian. The method reformulates inputs as fill-in-the-blank phrases, fine-tunes separate language models on a tiny labeled set (ranging from 0 to 1,000 examples), combines them to soft-label a large pool of unlabeled data, and finally distills this knowledge into a standard classifier. Experiments were conducted primarily using single GPUs with standard Transformer architectures.

The findings demonstrate substantial performance advantages for PET in limited-data scenarios. First, with just 10 labeled training examples, PET achieved accuracies of 52.9% on sentiment analysis and 87.5% on news classification, whereas standard supervised baselines barely performed above random chance at 21.1% and 25.0%, respectively. Second, under a completely zero-shot setup, iPET consistently outperformed standard unsupervised baselines across all tasks and even surpassed a supervised model trained on 1,000 examples in news classification. Third, PET and iPET consistently outperformed state-of-the-art semi-supervised methods that rely on back-translation, without requiring thousands of translation examples. Fourth, in multilingual testing, the approach yielded large performance improvements across German, French, and zero-shot Italian stance detection, maintaining gains even when training sets grew to thousands of examples.

These results indicate that organizations can build high-performing text classifiers at a fraction of standard data annotation costs and without massive computing infrastructure. By structuring tasks to leverage the inherent predictive capabilities of pretrained language models, teams can shorten project timelines and lower the barrier to entry for low-resource or domain-specific language tasks. The process of knowledge distillation also ensures the final deployed classifier remains standard in size and efficient for production inference.

Organizations operating in data-scarce environments should consider adopting cloze-style task reformulation over standard fine-tuning. When human pattern design is too time-consuming, teams can implement automated verbalizer search as a fallback, which the article showed still outperforms standard supervised training despite trailing handcrafted patterns. Where unlabeled data is plentiful, iterative training across multiple model generations is recommended to progressively improve label accuracy.

Confidence in these findings is high across the tested classification benchmarks, supported by consistent gains across multiple runs and hyperparameter configurations. However, the approach is bounded by the need to formulate tasks as single-token fill-in-the-blank problems and relies on the availability of relevant in-domain unlabeled text. Practitioners should carefully validate pattern selection and consider auxiliary language modeling objectives when working with extremely small sample sizes.

📚 What to read before this

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Jacob Devlin et al. (2019). Introduces the masked language modeling pre-training objective and Transformer architecture that Pattern-Exploiting Training relies upon for cloze-style reformulation.
RoBERTa: A Robustly Optimized BERT Pretraining Approach, Yinhan Liu et al. (2019). Provides the optimized masked language model underlying the primary empirical experiments and baseline evaluations in the source paper.
Language Models are Unsupervised Multitask Learners, Alec Radford et al. (2019). Demonstrates that language models can perform downstream tasks zero-shot via natural language task descriptions, motivating cloze-style prompt formulations.
Unsupervised Data Augmentation for Consistency Training, Qizhe Xie et al. (2020). Establishes semi-supervised consistency training baselines for text classification that Pattern-Exploiting Training explicitly compares against and seeks to outperform.
How to Fine-Tune BERT for Text Classification?, Chi Sun et al. (2019). Details standard fine-tuning strategies and failure modes of BERT on text classification, highlighting the low-resource limitations PET addresses.
Universal Language Model Fine-tuning for Text Classification, Jeremy Howard et al. (2018). Pioneers the transfer learning and fine-tuning methodology for text classification that serves as the foundation for few-shot adaptation techniques.

📚 What to read after this

Making Pre-trained Language Models Better Few-shot Learners, Tianyu Gao et al. (2021). Directly builds upon PET by introducing LM-BFF, which automates prompt and verbalizer generation and integrates demonstration sampling into few-shot fine-tuning.
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing, Pengfei Liu et al. (2021). Provides a comprehensive taxonomy and survey of prompting methods, placing cloze-based Pattern-Exploiting Training into the broader prompt-based learning paradigm.
Multitask Prompted Training Enables Zero-Shot Task Generalization, Victor Sanh et al. (2021). Scales the concept of prompted task reformulations to massive multitask mixture training (T0) to achieve robust zero-shot generalization across diverse held-out tasks.
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning, Haokun Liu et al. (2022). Extends few-shot prompt-based adaptation by developing parameter-efficient fine-tuning techniques (T-Few) that outperform in-context learning at lower cost.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?, Sewon Min et al. (2022). Analyzes the fundamental mechanisms behind prompt-based learning and in-context demonstrations to explain why prompted formulations succeed.
What Makes Good In-Context Examples for GPT-3?, Jiachang Liu et al. (2021). Investigates similarity-based exemplar retrieval to optimize the selection of demonstration prompts for few-shot learning.
Finetuned Language Models Are Zero-Shot Learners, Jason Wei et al. (2022). Generalizes task-instruction prompting through large-scale instruction fine-tuning to improve zero-shot and few-shot natural language understanding.

Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference

Timo Schick
Sulzer GmbH
Munich, Germany
[email protected]

Hinrich Schütze
Center for Information and Language Processing
LMU Munich, Germany
[email protected]

Abstract

Some NLP tasks can be solved in a fully unsupervised fashion by providing a pretrained language model with "task descriptions" in natural language (e.g., [1]). While this approach underperforms its supervised counterpart, we show in this work that the two ideas can be combined: We introduce Pattern-Exploiting Training ($\textsc{Pet}$), a semi-supervised training procedure that reformulates input examples as cloze-style phrases to help language models understand a given task. These phrases are then used to assign soft labels to a large set of unlabeled examples. Finally, standard supervised training is performed on the resulting training set. For several tasks and languages, $\textsc{Pet}$ outperforms supervised training and strong semi-supervised approaches in low-resource settings by a large margin.

Executive Summary: Applying natural language processing to real-world tasks often faces a severe bottleneck: annotating large volumes of high-quality data is expensive, time-consuming, and impractical across numerous languages and domains. While standard supervised learning performs poorly with only a handful of examples, pretrained language models possess substantial latent knowledge that typically goes underutilized unless prompted effectively.

This article set out to evaluate whether reformulating classification inputs into natural language cloze questions—termed Pattern-Exploiting Training (PET)—can combine the benefits of task descriptions with supervised fine-tuning to outperform conventional learning methods in low-resource settings.

The authors conducted an experimental evaluation comparing PET and an iterative extension (iPET) against standard supervised baselines, unsupervised prompting, and leading semi-supervised approaches. The evaluation covered four English datasets spanning sentiment analysis, news classification, question categorization, and natural language inference, as well as a multilingual stance detection dataset covering German, French, and Italian. The method reformulates inputs as fill-in-the-blank phrases, fine-tunes separate language models on a tiny labeled set (ranging from 0 to 1,000 examples), combines them to soft-label a large pool of unlabeled data, and finally distills this knowledge into a standard classifier. Experiments were conducted primarily using single GPUs with standard Transformer architectures.

The findings demonstrate substantial performance advantages for PET in limited-data scenarios. First, with just 10 labeled training examples, PET achieved accuracies of 52.9% on sentiment analysis and 87.5% on news classification, whereas standard supervised baselines barely performed above random chance at 21.1% and 25.0%, respectively. Second, under a completely zero-shot setup, iPET consistently outperformed standard unsupervised baselines across all tasks and even surpassed a supervised model trained on 1,000 examples in news classification. Third, PET and iPET consistently outperformed state-of-the-art semi-supervised methods that rely on back-translation, without requiring thousands of translation examples. Fourth, in multilingual testing, the approach yielded large performance improvements across German, French, and zero-shot Italian stance detection, maintaining gains even when training sets grew to thousands of examples.

These results indicate that organizations can build high-performing text classifiers at a fraction of standard data annotation costs and without massive computing infrastructure. By structuring tasks to leverage the inherent predictive capabilities of pretrained language models, teams can shorten project timelines and lower the barrier to entry for low-resource or domain-specific language tasks. The process of knowledge distillation also ensures the final deployed classifier remains standard in size and efficient for production inference.

Organizations operating in data-scarce environments should consider adopting cloze-style task reformulation over standard fine-tuning. When human pattern design is too time-consuming, teams can implement automated verbalizer search as a fallback, which the article showed still outperforms standard supervised training despite trailing handcrafted patterns. Where unlabeled data is plentiful, iterative training across multiple model generations is recommended to progressively improve label accuracy.

Confidence in these findings is high across the tested classification benchmarks, supported by consistent gains across multiple runs and hyperparameter configurations. However, the approach is bounded by the need to formulate tasks as single-token fill-in-the-blank problems and relies on the availability of relevant in-domain unlabeled text. Practitioners should carefully validate pattern selection and consider auxiliary language modeling objectives when working with extremely small sample sizes.

1. Introduction

Section Summary: In natural language processing, models are usually trained on labeled examples to learn a task, but with only a small number of examples this approach often fails because the task itself remains unclear. The section explains that adding a simple textual description of what the task involves makes it far easier to generalize correctly from few examples, and that pretrained language models can now make use of such descriptions. The authors introduce Pattern-Exploiting Training (PET), a method that turns inputs into fill-in-the-blank questions using these descriptions, fine-tunes several models, and then uses them to label additional data for training a final classifier, showing strong gains over standard supervised and semi-supervised methods in few-shot settings.

Learning from examples is the predominant approach for many NLP tasks: A model is trained on a set of labeled examples from which it then generalizes to unseen data. Due to the vast number of languages, domains and tasks and the cost of annotating data, it is common in real-world uses of NLP to have only a small number of labeled examples, making few-shot learning a highly important research area. Unfortunately, applying standard supervised learning to small training sets often performs poorly; many problems are difficult to grasp from just looking at a few examples. For instance, assume we are given the following pieces of text:

$T_1$: This was the best pizza I've ever had.
$T_2$: You can get better sushi for half the price.
$T_3$: Pizza was average. Not worth the price.

Furthermore, imagine we are told that the labels of $T_1$ and $T_2$ are $l$ and $l'$, respectively, and we are asked to infer the correct label for $T_3$. Based only on these examples, this is impossible because plausible justifications can be found for both $l$ and $l'$. However, if we know that the underlying task is to identify whether the text says anything about prices, we can easily assign $l'$ to $T_3$. This illustrates that solving a task from only a few examples becomes much easier when we also have a task description, i.e., a textual explanation that helps us understand what the task is about.

$**Figure 1:** $\textsc{Pet}$ for sentiment classification. **(1)** A number of patterns encoding some form of task description are created to convert training examples to cloze questions; for each pattern, a pretrained language model is finetuned. **(2)** The ensemble of trained models annotates unlabeled data. **(3)** A classifier is trained on the resulting soft-labeled dataset.$ {width=60%}

With the rise of pretrained language models (PLMs) such as GPT ([2]), BERT ([3]) and RoBERTa ([4]), the idea of providing task descriptions has become feasible for neural architectures: We can simply append such descriptions in natural language to an input and let the PLM predict continuations that solve the task ([1, 5]). So far, this idea has mostly been considered in zero-shot scenarios where no training data is available at all.

In this work, we show that providing task descriptions can successfully be combined with standard supervised learning in few-shot settings: We introduce Pattern-Exploiting Training ($\textsc{Pet}$), a semi-supervised training procedure that uses natural language patterns to reformulate input examples into cloze-style phrases. As illustrated in Figure 1, $\textsc{Pet}$ works in three steps: First, for each pattern a separate PLM is finetuned on a small training set $\mathcal{T}$. The ensemble of all models is then used to annotate a large unlabeled dataset $\mathcal{D}$ with soft labels. Finally, a standard classifier is trained on the soft-labeled dataset. We also devise i $\textsc{Pet}$, an iterative variant of $\textsc{Pet}$ in which this process is repeated with increasing training set sizes.

On a diverse set of tasks in multiple languages, we show that given a small to medium number of labeled examples, $\textsc{Pet}$ and i $\textsc{Pet}$ substantially outperform unsupervised approaches, supervised training and strong semi-supervised baselines.

2. Related Work

Section Summary: Recent research has explored using natural language hints, rephrased questions, or cloze-style prompts to help pretrained language models tackle tasks such as classification or question answering without any training examples, and to test what knowledge these models already possess. Other few-shot approaches draw on data from related tasks or generate additional examples through techniques like back-translation, though these often require extra labeled data or parallel corpora. The approach described here differs by needing no such resources and instead iteratively refines models in a self-training style, offering a straightforward way to incorporate human knowledge about the task.

[1] provide hints in the form of natural language patterns for zero-shot learning of challenging tasks such as reading comprehension and question answering (QA). This idea has been applied to unsupervised text classification ([5]), commonsense knowledge mining ([6]) and argumentative relation classification ([7]). [8] use task descriptions for zero-shot classification but require a semantic parser. For relation extraction, [9] automatically identify patterns that express given relations. [10] rephrase several tasks as QA problems. [11] frame various problems as language modeling tasks, but their patterns only loosely resemble natural language and are unsuitable for few-shot learning.[^1]

[^1]: For example, they convert inputs $(a,b)$ for recognizing textual entailment (RTE) to "rte sentence1: $a$ sentence2: $b

quot;, and the PLM is asked to predict strings like "not_entailment".

Another recent line of work uses cloze-style phrases to probe the knowledge that PLMs acquire during pretraining; this includes probing for factual and commonsense knowledge ([12, 13, 14, 15]), linguistic capabilities ([16, 17]), understanding of rare words ([18]), and ability to perform symbolic reasoning ([19]). [20] consider the problem of finding the best pattern to express a given task.

Other approaches for few-shot learning in NLP include exploiting examples from related tasks ([21, 22, 23, 24, 25]) and using data augmentation ([26, 27]); the latter commonly relies on back-translation ([28]), requiring large amounts of parallel data. Approaches using textual class descriptors typically assume that abundant examples are available for a subset of classes (e.g., [29, 30, 31]). In contrast, our approach requires no additional labeled data and provides an intuitive interface to leverage task-specific human knowledge.

The idea behind i $\textsc{Pet}$ – training multiple generations of models on data labeled by previous generations – bears resemblance to self-training and bootstrapping approaches for word sense disambiguation ([32]), relation extraction ([33, 34, 35]), parsing ([36, 37, 38]), machine translation ([39]), and sequence generation ([40]).

3. Pattern-Exploiting Training

Section Summary: Pattern-exploiting training turns a classification task into a fill-in-the-blank problem by wrapping each input in a short pattern that contains exactly one mask token and pairing it with a verbalizer that maps every label to a single vocabulary word. A pretrained language model is fine-tuned on a handful of labeled examples by predicting the correct verbalized word at the mask, while an auxiliary language-modeling loss on unlabeled inputs prevents the model from forgetting its original knowledge. Several such pattern-verbalizer pairs are trained separately, their predictions are combined to softly label a large unlabeled set, and the resulting dataset is used to train a conventional classifier that serves as the final output.

Let $M$ be a masked language model with vocabulary $V$ and mask token $____ \in V$, and let $\mathcal{L}$ be a set of labels for our target classification task $A$. We write an input for task $A$ as a sequence of phrases $\mathbf{x} = (s_1, \ldots, s_k)$ with $s_i \in V^*$; for example, $k=2$ if $A$ is textual inference (two input sentences). We define a pattern to be a function $P$ that takes $\mathbf{x}$ as input and outputs a phrase or sentence $P(\mathbf{x}) \in V^*$ that contains exactly one mask token, i.e., its output can be viewed as a cloze question. Furthermore, we define a verbalizer as an injective function $v: \mathcal{L} \rightarrow V$ that maps each label to a word from $M

#39;s vocabulary. We refer to $(P, v)$ as a pattern-verbalizer pair (PVP).

Using a PVP $(P,v)$ enables us to solve task $A$ as follows: Given an input $\mathbf{x}$, we apply $P$ to obtain an input representation $P(\mathbf{x})$, which is then processed by $M$ to determine the label $y \in \mathcal{L}$ for which $v(y)$ is the most likely substitute for the mask. For example, consider the task of identifying whether two sentences $a$ and $b$ contradict each other (label $y_0$) or agree with each other ($y_1$). For this task, we may choose the pattern $P(a,b) = \colorbox{#E8E8E8}{a? ____, b.}$ combined with a verbalizer $v$ that maps $y_0$ to "Yes" and $y_1$ to "No". Given an example input pair

$ \mathbf{x} = (\text{Mia likes pie, Mia hates pie}), $

the task now changes from having to assign a label without inherent meaning to answering whether the most likely choice for the masked position in

$ P(\mathbf{x}) = \colorbox{#E8E8E8}{Mia likes pie? ____, Mia hates pie.} $

is "Yes" or "No".

3.1 PVP Training and Inference

Let $\mathbf{p} = (P,v)$ be a PVP. We assume access to a small training set $\mathcal{T}$ and a (typically much larger) set of unlabeled examples $\mathcal{D}$. For each sequence $\mathbf{z} \in V^*$ that contains exactly one mask token and $w \in V$, we denote with $M(w \mid \mathbf{z})$ the unnormalized score that the language model assigns to $w$ at the masked position. Given some input $\mathbf{x}$, we define the score for label $l \in \mathcal{L}$ as

$ s_\mathbf{p}(l \mid \mathbf{x}) = M(v(l) \mid P(\textbf{x})) $

and obtain a probability distribution over labels using softmax:

$ q_\mathbf{p}(l \mid \mathbf{x}) = \frac{e^{s_\mathbf{p}(l \mid \mathbf{x})}}{ \sum_{l' \in \mathcal{L}} e^{s_\mathbf{p}(l' \mid \mathbf{x})}} $

We use the cross-entropy between $q_\mathbf{p}(l \mid \mathbf{x})$ and the true (one-hot) distribution of training example $(\mathbf{x}, l)$ – summed over all $(\mathbf{x}, l) \in \mathcal{T}$ – as loss for finetuning $M$ for $\mathbf{p}$.

3.2 Auxiliary Language Modeling

In our application scenario, only a few training examples are available and catastrophic forgetting can occur. As a PLM finetuned for some PVP is still a language model at its core, we address this by using language modeling as auxiliary task. With $L_\text{CE}$ denoting cross-entropy loss and $L_\text{MLM}$ language modeling loss, we compute the final loss as

$ L = (1 - \alpha) \cdot L_\text{CE} + \alpha \cdot L_\text{MLM} $

This idea was recently applied by [41] in a data-rich scenario. As $L_\text{MLM}$ is typically much larger than $L_\text{CE}$, in preliminary experiments, we found a small value of $\alpha = 10^{-4}$ to consistently give good results, so we use it in all our experiments. To obtain sentences for language modeling, we use the unlabeled set $\mathcal{D}$. However, we do not train directly on each $\mathbf{x} \in \mathcal{D}$, but rather on $P(\mathbf{x})$, where we never ask the language model to predict anything for the masked slot.

3.3 Combining PVPs

A key challenge for our approach is that in the absence of a large development set, it is hard to identify which PVPs perform well. To address this, we use a strategy similar to knowledge distillation ([42]). First, we define a set $\mathcal{P}$ of PVPs that intuitively make sense for a given task $A$. We then use these PVPs as follows:

(1) We finetune a separate language model $M_\mathbf{p}$ for each $\mathbf{p} \in \mathcal{P}$ as described in Section 3.1. As $\mathcal{T}$ is small, this finetuning is cheap even for a large number of PVPs.
(2) We use the ensemble $\mathcal{M} = { M_\mathbf{p} \mid \mathbf{p} \in \mathcal{P} }$ of finetuned models to annotate examples from $\mathcal{D}$. We first combine the unnormalized class scores for each example $\mathbf{x} \in \mathcal{D}$ as

$ {s}\mathcal{M}(l \mid \mathbf{x}) = \frac{1}{Z} \sum{\mathbf{p} \in \mathcal{P}} w(\mathbf{p}) \cdot s_\mathbf{p}(l \mid \mathbf{x}) $

where $Z = \sum_{\mathbf{p} \in \mathcal{P}} w(\mathbf{p})$ and the $w(\mathbf{p})$ are weighting terms for the PVPs.

We experiment with two different realizations of this weighing term: either we simply set $w(\mathbf{p}) = 1$ for all $\mathbf{p}$ or we set $w(\mathbf{p})$ to be the accuracy obtained using $\mathbf{p}$ on the training set before training. We refer to these two variants as uniform and weighted. [20] use a similar idea in a zero-shot setting.

We transform the above scores into a probability distribution $q$ using softmax. Following [42], we use a temperature of $T=2$ to obtain a suitably soft distribution. All pairs $(\mathbf{x}, q)$ are collected in a (soft-labeled) training set $\mathcal{T}_C$.

(3) We finetune a PLM $C$ with a standard sequence classification head on $\mathcal{T}_C$.

The finetuned model $C$ then serves as our classifier for $A$. All steps described above are depicted in Figure 2; an example is shown in Figure 1.

$**Figure 2:** Schematic representation of $\textsc{Pet}$ (1-3) and i $\textsc{Pet}$ (a-c). **(1)** The initial training set is used to finetune an ensemble of PLMs. **(a)** For each model, a random subset of other models generates a new training set by labeling examples from $\mathcal{D}$. **(b)** A new set of $\textsc{Pet}$ models is trained using the larger, model-specific datasets. **(c)** The previous two steps are repeated $k$ times, each time increasing the size of the generated training sets by a factor of $d$. **(2)** The final set of models is used to create a soft-labeled dataset $\mathcal{T}_C$. **(3)** A classifier $C$ is trained on this dataset.$

3.4 Iterative Pet (iPet)

Distilling the knowledge of all individual models into a single classifier $C$ means they cannot learn from each other. As some patterns perform (possibly much) worse than others, the training set $\mathcal{T}_C$ for our final model may therefore contain many mislabeled examples.

To compensate for this shortcoming, we devise i $\textsc{Pet}$, an iterative variant of $\textsc{Pet}$. The core idea of i $\textsc{Pet}$ is to train several generations of models on datasets of increasing size. To this end, we first enlarge the original dataset $\mathcal{T}$ by labeling selected examples from $\mathcal{D}$ using a random subset of trained $\textsc{Pet}$ models (Figure 2 a). We then train a new generation of $\textsc{Pet}$ models on the enlarged dataset (b); this process is repeated several times (c).

More formally, let $\mathcal{M}^0 = { M_1^0, \ldots, M_n^0 }$ be the initial set of $\textsc{Pet}$ models finetuned on $\mathcal{T}$, where each $M_i^0$ is trained for some PVP $\mathbf{p}_i$. We train $k$ generations of models $\mathcal{M}^1, \ldots, \mathcal{M}^k$ where $\mathcal{M}^j = { M_1^j, \ldots, M_n^j }$ and each $M_i^j$ is trained for $\mathbf{p}_i$ on its own training set $\mathcal{T}_i^j$. In each iteration, we multiply the training set size by a fixed constant $d \in \mathbb{N}$ while maintaining the label ratio of the original dataset. That is, with $c_0(l)$ denoting the number of examples with label $l$ in $\mathcal{T}$, each $\mathcal{T}i^j$ contains $c_j(l) = d \cdot c{j-1}(l)$ examples with label $l$. This is achieved by generating each $\mathcal{T}_i^j$ as follows:

We obtain $\mathcal{N} \subset \mathcal{M}^{j-1} \setminus { M_i^{j-1} }$ by randomly choosing $\lambda \cdot (n-1)$ models from the previous generation with $\lambda \in (0,1]$ being a hyperparameter.
Using this subset, we create a labeled dataset

$ {\mathcal{T}}\mathcal{N} = { (\mathbf{x}, \arg\max{l \in \mathcal{L}} s_\mathcal{N} (l \mid \mathbf{x})) \mid \mathbf{x} \in \mathcal{D} },. $

For each $l \in \mathcal{L}$, we obtain $\mathcal{T}\mathcal{N}(l) \subset \mathcal{T}\mathcal{N}$ by randomly choosing $c_j(l) - c_0(l)$ examples with label $l$ from $\mathcal{T}\mathcal{N}$. To avoid training future generations on mislabeled data, we prefer examples for which the ensemble of models is confident in its prediction. The underlying intuition is that even without calibration, examples for which labels are predicted with high confidence are typically more likely to be classified correctly ([43]). Therefore, when drawing from $\mathcal{T}\mathcal{N}$, we set the probability of each $(\mathbf{x}, y)$ proportional to $s_\mathcal{N}(l \mid \mathbf{x})$. 3. We define $\mathcal{T}i^j = \mathcal{T} \cup \bigcup{l \in \mathcal{L}} \mathcal{T}_\mathcal{N}(l)$. As can easily be verified, this dataset contains $c_j(l)$ examples for each $l \in \mathcal{L}$.

After training $k$ generations of $\textsc{Pet}$ models, we use $\mathcal{M}^k$ to create $\mathcal{T}_C$ and train $C$ as in basic $\textsc{Pet}$.

With minor adjustments, i $\textsc{Pet}$ can even be used in a zero-shot setting. To this end, we define $\mathcal{M}^0$ to be the set of untrained models and $c_1(l) = {10 / |\mathcal{L}|}$ for all $l \in \mathcal{L}$ so that $\mathcal{M}^1$ is trained on 10 examples evenly distributed across all labels. As $\mathcal{T}\mathcal{N}$ may not contain enough examples for some label $l$, we create all $\mathcal{T}\mathcal{N}(l)$ by sampling from the 100 examples $\mathbf{x} \in \mathcal{D}$ for which $s_\mathcal{N}(l \mid x)$ is the highest, even if $l \neq \arg\max_{l \in \mathcal{L}} s_\mathcal{N}(l \mid x)$. For each subsequent generation, we proceed exactly as in basic i $\textsc{Pet}$.

4. Experiments

Section Summary: The experiments evaluate the Pet method and its iterative variant on four English text classification datasets—Yelp reviews, AG's News, Yahoo Questions, and MNLI—plus a multilingual stance detection task, using RoBERTa large (or XLM-R) as the underlying language model. Performance is measured across varying few-shot training sizes with three random seeds per setting, relying on fixed hyperparameters such as a 1e-5 learning rate and 256-token maximum length rather than a large development set. Separate patterns and verbalizers are defined for each task to convert inputs and labels into natural language prompts suitable for the pretrained model.

We evaluate $\textsc{Pet}$ on four English datasets: Yelp Reviews, AG's News, Yahoo Questions ([44]) and MNLI ([45]). Additionally, we use x-stance ([46]) to investigate how well $\textsc{Pet}$ works for other languages. For all experiments on English, we use RoBERTa large ([4]) as language model; for x-stance, we use XLM-R ([47]). We investigate the performance of $\textsc{Pet}$ and all baselines for different training set sizes; each model is trained three times using different seeds and average results are reported.

As we consider a few-shot setting, we assume no access to a large development set on which hyperparameters could be optimized. Our choice of hyperparameters is thus based on choices made in previous work and practical considerations. We use a learning rate of $1\cdot10^{-5}$, a batch size of $16$ and a maximum sequence length of $256$. Unless otherwise specified, we always use the weighted variant of $\textsc{Pet}$ with auxiliary language modeling. For i $\textsc{Pet}$, we set $\lambda = 0.25$ and $d = 5$; that is, we select $25%$ of all models to label examples for the next generation and quintuple the number of training examples in each iteration. We train new generations until each model was trained on at least $1000$ examples, i.e., we set $k = \lceil \log_d(1000 / |\mathcal{T}|) \rceil$. As we always repeat training three times, the ensemble $\mathcal{M}$ (or $\mathcal{M}^0$) for $n$ PVPs contains $3n$ models. Further hyperparameters and detailed explanations for all our choices are given in Appendix B.

4.1 Patterns

We now describe the patterns and verbalizers used for all tasks. We use two vertical bars ($|$) to mark boundaries between text segments.[^2]

[^2]: The way different segments are handled depends on the model being used; they may e.g. be assigned different embeddings ([3]) or separated by special tokens ([4, 48]). For example, "$a$ $|$ $b

quot; is given to BERT as the input "[CLS] $a$ [SEP] $b$ [SEP]".

Yelp

For the Yelp Reviews Full Star dataset ([44]), the task is to estimate the rating that a customer gave to a restaurant on a $1$- to $5$-star scale based on their review's text. We define the following patterns for an input text $a$:

$ \begin{aligned} P_1(a) & = \colorbox{#E8E8E8}{It was ____. a} & P_2(a) & = \colorbox{#E8E8E8}{Just ____! \textbardbl a} \ P_3(a) & = \colorbox{#E8E8E8}{a. All in all, it was ____.}\ P_4(a) & = \colorbox{#E8E8E8}{a \textbardbl In summary, the restaurant is ____.} \end{aligned} $

We define a single verbalizer $v$ for all patterns as

$ \begin{aligned} v(1) & = \text{terrible} & v(2) & = \text{bad} & v(3) & = \text{okay} \ v(4) & = \text{good} & v(5) & = \text{great}, \end{aligned} $

AG's News

AG's News is a news classification dataset, where given a headline $a$ and text body $b$, news have to be classified as belonging to one of the categories World ($1$), Sports ($2$), Business ($3$) or Science/Tech ($4$). For $\mathbf{x} = (a,b)$, we define the following patterns:

$ \begin{aligned} P_1(\mathbf{x}) & = \colorbox{#E8E8E8}{____: a b} & P_2(\mathbf{x}) & = \colorbox{#E8E8E8}{a ( ____ ) b} \ P_3(\mathbf{x}) & = \colorbox{#E8E8E8}{____ -- a b} & P_4(\mathbf{x}) & = \colorbox{#E8E8E8}{a b ( ____ )} \ P_5(\mathbf{x}) & = \colorbox{#E8E8E8}{____ News: a b} \ P_6(\mathbf{x}) & = \colorbox{#E8E8E8}{[ Category: ____ ] a b} \end{aligned} $

We use a verbalizer that maps $1$–$4$ to "World", "Sports", "Business" and "Tech", respectively.

Yahoo

Yahoo Questions ([44]) is a text classification dataset. Given a question $a$ and an answer $b$, one of ten possible categories has to be assigned. We use the same patterns as for AG's News, but we replace the word "News" in $P_5$ with the word "Question". We define a verbalizer that maps categories $1$–$10$ to "Society", "Science", "Health", "Education", "Computer", "Sports", "Business", "Entertainment", "Relationship" and "Politics".

MNLI

The MNLI dataset ([45]) consists of text pairs $\mathbf{x} = (a,b)$. The task is to find out whether $a$ implies $b$ ($0$), $a$ and $b$ contradict each other ($1$) or neither ($2$). We define

$ \begin{aligned} P_1(\mathbf{x}),{=},\colorbox{#E8E8E8}{"a"? \textbardbl ____, "b"} && P_2(\mathbf{x}),{=},\colorbox{#E8E8E8}{a? \textbardbl ____, b} \end{aligned} $

and consider two different verbalizers $v_1$ and $v_2$:

$ \begin{alignedat}{3} v_1(0) & = \text{Wrong}\ \ & v_1(1) & = \text{Right}\ \ & v_1(2) & = \text{Maybe}\ v_2(0) & = \text{No} & v_2(1) & = \text{Yes} & v_2(2) & = \text{Maybe} \end{alignedat} $

Combining the two patterns with the two verbalizers results in a total of 4 PVPs.

\begin{tabularx}{0.87\linewidth}{cll>{\arraybackslash}X>{\arraybackslash}X>{\arraybackslash}Xc}
\toprule
\textbf{Line}&\textbf{Examples} & \textbf{Method} & \multicolumn{1}{c}{\textbf{Yelp}} & \multicolumn{1}{c}{\textbf{AG's}} & \multicolumn{1}{c}{\textbf{Yahoo}} & {\textbf{MNLI (m/mm)}} \\
\midrule
\phantom{1}1&\multirow{3}{*}{$|\mathcal{T}| = 0$} 
 {} &
unsupervised
(avg) & 33.8 \scriptsize $\pm$9.6 & 69.5 \scriptsize $\pm$7.2 & 44.0 \scriptsize $\pm$9.1 & 39.1 {\scriptsize $\pm$4.3} / 39.8 \scriptsize $\pm$5.1 \\
\phantom{1}2&&
unsupervised
(max) & 40.8 \scriptsize $\pm$0.0 & 79.4 \scriptsize $\pm$0.0 & 56.4 \scriptsize $\pm$0.0 & 43.8 {\scriptsize $\pm$0.0} / 45.0 \scriptsize $\pm$0.0 \\
\phantom{1}3&& i\textsc{Pet} & {\fontseries{b}\selectfont 56.7} \scriptsize $\pm$0.2 & {\fontseries{b}\selectfont 87.5} \scriptsize $\pm$0.1 & {\fontseries{b}\selectfont 70.7} \scriptsize $\pm$0.1 & {\fontseries{b}\selectfont 53.6} {\scriptsize $\pm$0.1} / {\fontseries{b}\selectfont 54.2} \scriptsize $\pm$0.1 \\
\midrule

\phantom{1}4&\multirow{3}{*}{$|\mathcal{T}| = 10$}
 {} & supervised & 21.1 \scriptsize $\pm$1.6 & 25.0 \scriptsize $\pm$0.1 & 10.1 \scriptsize $\pm$0.1 & 34.2 {\scriptsize $\pm$2.1} / 34.1 \scriptsize $\pm$2.0 \\
\phantom{1}5&& \textsc{Pet} & 52.9 \scriptsize $\pm$0.1 & 87.5 \scriptsize $\pm$0.0 & 63.8 \scriptsize $\pm$0.2 & 41.8 {\scriptsize $\pm$0.1} / 41.5 \scriptsize $\pm$0.2 \\
\phantom{1}6&& i\textsc{Pet} & {\fontseries{b}\selectfont 57.6} \scriptsize $\pm$0.0 & {\fontseries{b}\selectfont 89.3} \scriptsize $\pm$0.1 & {\fontseries{b}\selectfont 70.7} \scriptsize $\pm$0.1 & {\fontseries{b}\selectfont 43.2} {\scriptsize $\pm$0.0} / {\fontseries{b}\selectfont 45.7} \scriptsize $\pm$0.1 \\
\midrule

\phantom{1}7&\multirow{3}{*}{$|\mathcal{T}| = 50$}
 {} & supervised & 44.8 \scriptsize $\pm$2.7 & 82.1 \scriptsize $\pm$2.5 & 52.5 \scriptsize $\pm$3.1 & 45.6 {\scriptsize $\pm$1.8} / 47.6 \scriptsize $\pm$2.4 \\
\phantom{1}8&& \textsc{Pet} & 60.0 \scriptsize $\pm$0.1 & 86.3 \scriptsize $\pm$0.0 & 66.2 \scriptsize $\pm$0.1 & 63.9 {\scriptsize $\pm$0.0} / 64.2 \scriptsize $\pm$0.0 \\
\phantom{1}9&& i\textsc{Pet} & {\fontseries{b}\selectfont 60.7} \scriptsize $\pm$0.1 & {\fontseries{b}\selectfont 88.4} \scriptsize $\pm$0.1 & {\fontseries{b}\selectfont 69.7} \scriptsize $\pm$0.0 & {\fontseries{b}\selectfont 67.4} {\scriptsize $\pm$0.3} / {\fontseries{b}\selectfont 68.3} \scriptsize $\pm$0.3 \\
\midrule

10&\multirow{3}{*}{$|\mathcal{T}| = 100$}
 {} & supervised & 53.0 \scriptsize $\pm$3.1 & 86.0 \scriptsize $\pm$0.7 & 62.9 \scriptsize $\pm$0.9 & 47.9 {\scriptsize $\pm$2.8} / 51.2 \scriptsize $\pm$2.6 \\
11&& \textsc{Pet} & 61.9 \scriptsize $\pm$0.0 & 88.3 \scriptsize $\pm$0.1 & 69.2 \scriptsize $\pm$0.0 & 74.7 {\scriptsize $\pm$0.3} / 75.9 \scriptsize $\pm$0.4 \\
12&& i\textsc{Pet} & {\fontseries{b}\selectfont 62.9} \scriptsize $\pm$0.0 & {\fontseries{b}\selectfont 89.6} \scriptsize $\pm$0.1 & {\fontseries{b}\selectfont 71.2} \scriptsize $\pm$0.1 & {\fontseries{b}\selectfont 78.4} {\scriptsize $\pm$0.7} / {\fontseries{b}\selectfont 78.6} \scriptsize $\pm$0.5 \\
\midrule

13&\multirow{2}{*}{$|\mathcal{T}| = 1000$}
 {} & supervised & 63.0 \scriptsize $\pm$0.5 & {\fontseries{b}\selectfont 86.9} \scriptsize $\pm$0.4 & 70.5 \scriptsize $\pm$0.3 & 73.1 {\scriptsize $\pm$0.2} / 74.8 \scriptsize $\pm$0.3 \\
14&& \textsc{Pet} & {\fontseries{b}\selectfont 64.8} \scriptsize $\pm$0.1 & {\fontseries{b}\selectfont 86.9} \scriptsize $\pm$0.2 & {\fontseries{b}\selectfont 72.7} \scriptsize $\pm$0.0 & {\fontseries{b}\selectfont 85.3} {\scriptsize $\pm$0.2} / {\fontseries{b}\selectfont 85.5} \scriptsize $\pm$0.4 \\
\bottomrule
\end{tabularx}

X-Stance

The x-stance dataset ([46]) is a multilingual stance detection dataset with German, French and Italian examples. Each example $\mathbf{x} = (a,b)$ consists of a question $a$ concerning some political issue and a comment $b$; the task is to identify whether the writer of $b$ supports the subject of the question ($0$) or not ($1$). We use two simple patterns

$ \begin{aligned} P_1(\mathbf{x}) = \colorbox{#E8E8E8}{"a" \textbardbl ____. "b"} && P_2(\mathbf{x}) = \colorbox{#E8E8E8}{a \textbardbl ____. b} \end{aligned} $

and define an English verbalizer $v_\text{En}$ mapping $0$ to "Yes" and $1$ to "No" as well as a French (German) verbalizer $v_\text{Fr}$ ($v_\text{De}$), replacing "Yes" and "No" with "Oui" and "Non" ("Ja" and "Nein"). We do not define an Italian verbalizer because x-stance does not contain any Italian training examples.

4.2 Results

English Datasets

Table 1 shows results for English text classification and language understanding tasks; we report mean accuracy and standard deviation for three training runs. Lines 1–2 (L1–L2) show unsupervised performance, i.e., individual PVPs without any training (similar to [2, 5]); we give both average results across all PVPs (avg) and results for the PVP that works best on the test set (max). The large difference between both rows highlights the importance of coping with the fact that without looking at the test set, we have no means of evaluating which PVPs perform well. Zero-shot i $\textsc{Pet}$ clearly outperforms the unsupervised baselines for all datasets (L3 vs L1); on AG's News, it even performs better than standard supervised training with 1000 examples (L3 vs L13). With just 10 training examples, standard supervised learning does not perform above chance (L4). In contrast, $\textsc{Pet}$ (L5) performs much better than the fully unsupervised baselines (L1–L2); training multiple generations using i $\textsc{Pet}$ (L6) gives consistent improvements. As we increase the training set size, the performance gains of $\textsc{Pet}$ and i $\textsc{Pet}$ become smaller, but for both 50 and 100 examples, $\textsc{Pet}$ continues to considerably outperform standard supervised training (L8 vs L7, L11 vs L10) with i $\textsc{Pet}$ (L9, L12) still giving consistent improvements. For $|\mathcal{T}| = 1000$, $\textsc{Pet}$ has no advantage on AG's but still improves accuracy for all other tasks (L14 vs L13) .[^3]

[^3]: One of the three supervised MNLI runs for $|\mathcal{T}|=1000$ underfitted the training data and performed extremely poorly. This run is excluded in the reported score (73.1/74.8).

\begin{tabularx}{\linewidth}{ll>{\arraybackslash}X>{\arraybackslash}X>{\arraybackslash}Xc}
        \toprule
        \textbf{Ex.} & \textbf{Method} & \multicolumn{1}{c}{\textbf{Yelp}} & \multicolumn{1}{c}{\textbf{AG's}} & \multicolumn{1}{c}{\textbf{Yahoo}} & \multicolumn{1}{c}{\textbf{MNLI}} \\
        \midrule
        \multirow{4}{*}{\rotatebox[origin=c]{90}{$|\mathcal{T}| = 10$}} 
         {} & UDA & 27.3 & 72.6 & 36.7 & 34.7 \\
         {} & MixText & 20.4 & 81.1 & 20.6 & 32.9 \\
         {} & \textsc{Pet} & 48.8 & 84.1 & 59.0 & 39.5 \\
         {} & i\textsc{Pet} & \fontseries{b}\selectfont 52.9 & \fontseries{b}\selectfont 87.5 & \fontseries{b}\selectfont 67.0 & \fontseries{b}\selectfont 42.1 \\
        \midrule
        \multirow{4}{*}{\rotatebox[origin=c]{90}{$|\mathcal{T}| = 50$}} 
         {} & UDA & 46.6 & 83.0 & 60.2 & 40.8 \\
      {} & MixText & 31.3 & 84.8 & 61.5 & 34.8 \\
      {} & \textsc{Pet} & 55.3 & 86.4 & 63.3 & 55.1 \\
         {} & i\textsc{Pet} & \fontseries{b}\selectfont 56.7 & \fontseries{b}\selectfont 87.3 & \fontseries{b}\selectfont 66.4 & \fontseries{b}\selectfont 56.3 \\  
        \bottomrule
    \end{tabularx}

Comparison with SotA

We compare $\textsc{Pet}$ to UDA ([26]) and MixText ([27]), two state-of-the-art methods for semi-supervised learning in NLP that rely on data augmentation. Whereas $\textsc{Pet}$ requires that a task can be expressed using patterns and that such patterns be found, UDA and MixText both use backtranslation ([28]) and thus require thousands of labeled examples for training a machine translation model. We use RoBERTa (base) for our comparison as MixText is specifically tailored towards a 12-layer Transformer ([49]). Both [26] and [27] use large development sets to optimize the number of training steps. We instead try several values for both approaches directly on the test set and only report the best results obtained. Despite this, Table 2 shows that $\textsc{Pet}$ and i $\textsc{Pet}$ substantially outperform both methods across all tasks, clearly demonstrating the benefit of incorporating human knowledge in the form of PVPs.

X-Stance

We evaluate $\textsc{Pet}$ on x-stance to investigate (i) whether it works for languages other than English and (ii) whether it also brings improvements when training sets have medium size. In contrast to [46], we do not perform any hyperparameter optimization on dev and use a shorter maximum sequence length (256 vs 512) to speed up training and evaluation.

::: {caption="Table 3: Results on x-stance intra-target for XLM-R (base) trained on subsets of $\mathcal{T}\text{De}$ and $\mathcal{T}\text{Fr}$ and for joint training on all data ($\mathcal{T}\text{De} + \mathcal{T}\text{Fr}$). (*): Best results for mBERT reported in [46]."}

:::

To investigate whether $\textsc{Pet}$ brings benefits even when numerous examples are available, we consider training set sizes of $1000$, $2000$, and $4000$; for each of these configurations, we separately finetune French and German models to allow for a more straightforward downsampling of the training data. Additionally, we train models on the entire French ($|\mathcal{T}\text{Fr}| = 11,790$) and German ($|\mathcal{T}\text{De}| = 33,850$) training sets. In this case we do not have any additional unlabeled data, so we simply set $\mathcal{D} = \mathcal{T}$. For the French models, we use $v_\text{En}$ and $v_\text{Fr}$ as verbalizers and for German $v_\text{En}$ and $v_\text{De}$ (Section 4.1). Finally, we also investigate the performance of a model trained jointly on French and German data ($|\mathcal{T}\text{Fr} + \mathcal{T}\text{De}| = 45,640$) using $v_\text{En}$, $v_\text{Fr}$ and $v_\text{De}$.

Results are shown in Table 3; following [46], we report the macro-average of the F1 scores for labels 0 and 1, averaged over three runs. For Italian (column "It"), we report the average zero-shot cross-lingual performance of German and French models as there are no Italian training examples. Our results show that $\textsc{Pet}$ brings huge improvements across all languages even when training on much more than a thousand examples; it also considerably improves zero-shot cross-lingual performance.

5. Analysis

Section Summary: The analysis shows that PET effectively combines multiple pattern-verbalizer pairs even when some perform poorly, outperforming the best individual pattern while knowledge distillation further boosts results and shrinks the final model. Adding an auxiliary language modeling objective helps most when training data is extremely limited, though its value drops as more examples become available. Iterative PET steadily improves accuracy over successive generations by gradually expanding the labeled set, and while in-domain pretraining aids both supervised baselines and PET, the method’s gains clearly stem from more than just extra unlabeled data.

\begin{tabularx}{\linewidth}{l>{\arraybackslash}X>{\arraybackslash}X>{\arraybackslash}Xc}
        \toprule
        \textbf{Method} & \textbf{Yelp} & \textbf{AG's} & \textbf{Yahoo} & \textbf{MNLI} \\
        \midrule
        min & 39.6 & 82.1 & 50.2 & 36.4 \\
        max & 52.4 & 85.0 & 63.6 & 40.2 \\
        \textsc{Pet} (no distillation) & 51.7 & 87.0 & 62.8 & 40.6 \\
        \textsc{Pet} uniform & 52.7 & 87.3 & \textbf{63.8} & \textbf{42.0} \\
        \textsc{Pet} weighted & \textbf{52.9} & \textbf{87.5} & \textbf{63.8} & 41.8 \\
        \bottomrule
    \end{tabularx}

$**Figure 3:** Accuracy improvements for $\textsc{Pet}$ due to adding $L_\text{MLM}$ during training$ {width=70%}

Combining PVPs

We first investigate whether $\textsc{Pet}$ is able to cope with situations were some PVPs perform much worse than others. For $|\mathcal{T}| = 10$, Table 4 compares the performance of $\textsc{Pet}$ to that of the best and worst performing patterns after finetuning; we also include results obtained using the ensemble of $\textsc{Pet}$ models corresponding to individual PVPs without knowledge distillation. Even after finetuning, the gap between the best and worst pattern is large, especially for Yelp. However, $\textsc{Pet}$ is not only able to compensate for this, but even improves accuracies over using only the best-performing pattern across all tasks. Distillation brings consistent improvements over the ensemble; additionally, it significantly reduces the size of the final classifier. We find no clear difference between the uniform and weighted variants of $\textsc{Pet}$.

$**Figure 4:** Average accuracy for each generation of models with i $\textsc{Pet}$ in a zero-shot setting. Accuracy on AG's News and Yahoo when skipping generation 2 and 3 is indicated through dashed lines.$ {width=70%}

Auxiliary Language Modeling

We analyze the influence of the auxiliary language modeling task on $\textsc{Pet}$ 's performance. Figure 3 shows performance improvements from adding the language modeling task for four training set sizes. We see that the auxiliary task is extremely valuable when training on just 10 examples. With more data, it becomes less important, sometimes even leading to worse performance. Only for MNLI, we find language modeling to consistently help.

Iterative Pet

To check whether i $\textsc{Pet}$ is able to improve models over multiple generations, Figure 4 shows the average performance of all generations of models in a zero-shot setting. Each additional iteration does indeed further improve the ensemble's performance. We did not investigate whether continuing this process for even more iterations gives further improvements.

Another natural question is whether similar results can be obtained with fewer iterations by increasing the training set size more aggressively. To answer this question, we skip generations 2 and 3 for AG's News and Yahoo and for both tasks directly let ensemble $\mathcal{M}^1$ annotate $10\cdot5^4$ examples for $\mathcal{M}^4$. As indicated in Figure 4 through dashed lines, this clearly leads to worse performance, highlighting the importance of only gradually increasing the training set size. We surmise that this is the case because annotating too many examples too early leads to a large percentage of mislabeled training examples.

In-Domain Pretraining

Unlike our supervised baseline, $\textsc{Pet}$ makes use of the additional unlabeled dataset $\mathcal{D}$. Thus, at least some of $\textsc{Pet}$ 's performance gains over the supervised baseline may arise from this additional in-domain data.

To test this hypothesis, we simply further pretrain RoBERTa on in-domain data, a common technique for improving text classification accuracy (e.g., [50, 51]). As language model pretraining is expensive in terms of GPU usage, we do so only for the Yelp dataset. Figure 5 shows results of supervised learning and $\textsc{Pet}$ both with and without this in-domain pretraining. While pretraining does indeed improve accuracy for supervised training, the supervised model still clearly performs worse than $\textsc{Pet}$, showing that the success of our method is not simply due to the usage of additional unlabeled data. Interestingly, in-domain pretraining is also helpful for $\textsc{Pet}$, indicating that $\textsc{Pet}$ leverages unlabeled data in a way that is clearly different from standard masked language model pretraining.

$**Figure 5:** Accuracy of supervised learning (sup.) and $\textsc{Pet}$ both with and without pretraining (PT) on Yelp$

6. Conclusion

Section Summary: The researchers showed that task descriptions can be effectively combined with pretrained language models and standard supervised training. Their PET method uses simple question-style patterns and word mappings to tap into the models' built-in knowledge, generating large labeled datasets that are then used to train ordinary classifiers. This approach delivers strong gains over regular training and other semi-supervised methods, especially when labeled data is scarce.

We have shown that providing task descriptions to pretrained language models can be combined with standard supervised training. Our proposed method, $\textsc{Pet}$, consists of defining pairs of cloze question patterns and verbalizers that help leverage the knowledge contained within pretrained language models for downstream tasks. We finetune models for all pattern-verbalizer pairs and use them to create large annotated datasets on which standard classifiers can be trained. When the initial amount of training data is limited, $\textsc{Pet}$ gives large improvements over standard supervised training and strong semi-supervised approaches.

Acknowledgments

This work was funded by the European Research Council (ERC #740516). We would like to thank the anonymous reviewers for their helpful comments.

Appendix

Section Summary: The appendix outlines the technical setup for implementing PET and iPET, which relies on standard libraries like Transformers and PyTorch, and runs most experiments on a single 11GB GPU. It explains the rationale behind key hyperparameter selections, such as batch sizes that fit in memory, reduced learning rates for stability, adjusted training steps to balance labeled and unlabeled data, and specific choices for ensembles, dataset growth in iPET iterations, and auxiliary losses. The section also reports model parameter counts, typical runtimes for training and labeling, and adjustments made for comparisons with methods like UDA and MixText.

A. Implementation

Our implementation of $\textsc{Pet}$ and i $\textsc{Pet}$ is based on the Transformers library ([52]) and PyTorch ([53]).

B. Training Details

Except for the in-domain pretraining experiment described in Section 5 , all of our experiments were conducted using a single GPU with 11GB RAM (NVIDIA GeForce GTX 1080 Ti).

B.1 Hyperparameter Choices

Relevant training hyperparameters for both individual $\textsc{Pet}$ models and the final classifier $C$ as well as our supervised baseline are listed in Table 5. All hyperparameters were selected based on the following considerations and experiments:

Batch size / maximum length

Both batch size and maximum sequence length (or block size) are chosen so that one batch fits into 11GB of GPU memory. As [3] and [4] use larger batch sizes of 16–32, we accumulate gradients for 4 steps to obtain an effective batch size of 16.

Learning rate

We found a learning rate of $5\mathrm{e}{-5}$ (as used by [3]) to often result in unstable training for regular supervised learning with no accuracy improvements on the training set. We therefore use a lower learning rate of $1\mathrm{e}{-5}$, similar to [4]. Experiments with various learning rates can be found in Appendix D.

Training steps

As the number of training epochs recommended by [4] in a data-rich scenario is in the range 2–10, we perform supervised training for 250 training steps, corresponding to 4 epochs when training on 1000 examples. For individual $\textsc{Pet}$ models, we subdivide each batch into one labeled example from $\mathcal{T}$ to compute $L_\text{CE}$ and three unlabeled examples from $\mathcal{D}$ to compute $L_\text{MLM}$. Accordingly, we multiply the number of total training steps by $4$ (i.e., 1000), so that the number of times each labeled example is seen remains constant ($16\cdot 250 = 4 \cdot 1000$). For the final $\textsc{Pet}$ classifier, we train for 5000 steps due to the increased training set size (depending on the task, the unlabeled set $\mathcal{D}$ contains at least $20,000$ examples). Deviating from the above, we always perform training for 3 epochs on x-stance to match the setup of [46] more closely. The effect of varying the number of training steps is further investigated in Appendix D.

Temperature

We choose a temperature of 2 when training the final classifier following [42].

Auxiliary language modeling

To find a suitable value of $\alpha$ for combining language modeling loss and cross-entropy loss, we first observed that in the early stages of training, the former is a few orders of magnitude higher than the latter for all tasks considered. We thus selected a range ${ 1\mathrm{e}{-}3, 1\mathrm{e}{-}4, 1\mathrm{e}{-}5 }$ of reasonable choices for $\alpha$ and performed preliminary experiments on Yelp with 100 training examples to find the best value among these candidates. To this end, we split the training examples into a training set and a dev set using both a 90/10 split and a 50/50 split and took the value of $\alpha$ that maximizes average dev set accuracy. We adopt this value for all other tasks and training set sizes without further optimization.

Models per ensemble

As we always train three models per pattern, for both i $\textsc{Pet}$ and training the final classifier $C$, the ensemble $\mathcal{M}$ (or $\mathcal{M}^0$) for $n$ PVPs contains $3n$ models. This ensures consistency as randomly choosing any of the three models for each PVP would result in high variance. In preliminary experiments, we found this to have only little impact on the final model's performance.

iPet dataset size

For i $\textsc{Pet}$, we quintuple the number of training examples after each iteration ($d = 5$) so that only a small number of generations is required to reach a sufficient amount of labeled data. We did not choose a higher value because we presume that this may cause training sets for early generations to contain a prohibitively large amount of mislabeled data.

iPet dataset creation

We create training sets for the next generation in i $\textsc{Pet}$ using 25% of the models in the current generation ($\lambda = 0.25$) because we want the training sets for all models to be diverse while at the same time, a single model should not have too much influence.

Others

For all other hyperparameters listed in Table 5, we took the default settings of the Transformers library ([52]).

B.2 Number of parameters

As $\textsc{Pet}$ does not require any additional learnable parameters, the number of parameters for both $\textsc{Pet}$ and i $\textsc{Pet}$ is identical to the number of parameters in the underlying language model: 355M for RoBERTa (large) and 270M for XLM-R (base).

B.3 Average runtime

Training a single $\textsc{Pet}$ classifier for 250 steps on one GPU took approximately 30 minutes; training for 1000 steps with auxiliary language modeling took $~$60 minutes. Depending on the task, labeling examples from $\mathcal{D}$ took 15–30 minutes per model. Training the final classifier $C$ for 5000 steps on the soft-labeled dataset $\mathcal{T}_C$ took 2 hours on average.

B.4 Comparison with SotA

For comparing $\textsc{Pet}$ to UDA ([26]) and MixText ([27]), we reduce the number of unlabeled examples by half to speed up the required backtranslation step. We use the backtranslation script provided by [27] with their recommended hyperparameter values and use both Russian and German as intermediate languages.

For MixText, we use the original implementation^4 and the default set of hyperparameters. Specifically, each batch consists of 4 labeled and 8 unlabeled examples, we use layers 7, 9 and 12 for mixing, we set $T=5$, $\alpha=16$, and use a learning rate of $5 \cdot 10^{-6}$ for RoBERTa and $5 \cdot 10 ^{-4}$ for the final classification layer. We optimize the number of training steps for each task and dataset size in the range ${1000, 2000, 3000, 4000, 5000}$.

For UDA, we use a PyTorch-based reimplementation^5. We use the same batch size as for MixText and the hyperparameter values recommended by [26]; we use an exponential schedule for training signal annealing and a learning rate of $2\cdot10^{-5}$. We optimize the number of training steps for each task and dataset size in the range ${500, 1000, 1500, \ldots, 10000}$.

B.5 In-Domain Pretraining

For in-domain pretraining experiments described in Section 5 , we use the language model finetuning script of the Transformers library ([52]); all hyperparameters are listed in the last column of Table 5. Pretraining was performed on a total of 3 NVIDIA GeForce GTX 1080 Ti GPUs.

\begin{tabularx}{\linewidth}{lXXXXX}
        \toprule
        \textbf{Parameter} & \textsc{Pet} $-$LM & \textsc{Pet} (En/Xs) & $C$ (En/Xs) & sup. (En/Xs) & In-Dom. PT \\
        \midrule
        \texttt{adam\_epsilon} & 1e-8 & 1e-8 & 1e-8 & 1e-8 & 1e-8 \\
        * \texttt{alpha} & – & 1e-4 & – & – & – \\
        \texttt{block\_size} & – & – & – & – & 256\\
        \texttt{gradient\_accumulation\_steps} & 4 & 4 & 4 & 4 & 2 \\
        \texttt{learning\_rate} & 1e-5 & 1e-5 & 1e-5 & 1e-5 & 5e-5 \\
        \texttt{max\_grad\_norm} & 1.0 & 1.0 & 1.0 & 1.0 & 1.0 \\
        \texttt{max\_seq\_length} & 256 & 256 & 256 & 256 & – \\
        \texttt{max\_steps} & 250 & 1000 / – & 5000 / – & 250 / – & 50000 \\
        \texttt{mlm\_probability} & – & 0.15 & – & – & 0.15 \\
        \texttt{num\_train\_epochs} & – & – / 3 & – / 3 & – / 3 & – \\
        \texttt{per\_gpu\_train\_batch\_size} & 4 & 1 & 4 & 4 & 2 \\
        * \texttt{per\_gpu\_helper\_batch\_size} & – & 3 & – & – & – \\
        * \texttt{temperature} & – & – & 2.0 & – & – \\
        \texttt{weight\_decay} & 0.01 & 0.01 & 0.01 & 0.01 & 0.0 \\
        \bottomrule
    \end{tabularx}

C. Dataset Details

For each task and number of examples $t$, we create the training set $\mathcal{T}$ by collecting the first $t/|\mathcal{L}|$ examples per label from the original training set, where $|\mathcal{L}|$ is the number of labels for the task. Similarly, we construct the set $\mathcal{D}$ of unlabeled examples by selecting $10,000$ examples per label and removing all labels. For evaluation, we use the official test set for all tasks except MNLI, for which we report results on the dev set; this is due to the limit of 2 submissions per 14 hours for the official MNLI test set. An overview of the number of test examples and links to downloadable versions of all used datasets can be found in Table 6.

Preprocessing

In some of the datasets used, newlines are indicated through the character sequence "\n". As the vocabularies of RoBERTa and XLM-R do not feature a newline, we replace this sequence with a single space. We do not perform any other preprocessing, except shortening all examples to the maximum sequence length of 256 tokens. This is done using the longest first strategy implemented in the Transformers library. For $\textsc{Pet}$, all input sequences are truncated before applying patterns.

Evaluation metrics

For Yelp, AG's News, Yahoo and MNLI, we use accuracy. For x-stance, we report macro-average of F1 scores using the evaluation script of [46].

\begin{tabularx}{\linewidth}{lXr}
        \toprule
        \textbf{Dataset} & \textbf{Link} & \textbf{Test Examples} \\
        \midrule
        AG's News & \url{http://goo.gl/JyCnZq} & 7600 \\
        MNLI (m / mm) & \url{https://cims.nyu.edu/~sbowman/multinli/} & 10000 / 10000 \\
        X-Stance (De / Fr / It) & \url{https://github.com/ZurichNLP/xstance} & 3479 / 1284 / 1173 \\
        Yahoo! Answers & \url{http://goo.gl/JyCnZq} & 60000 \\
        Yelp Review Full & \url{http://goo.gl/JyCnZq} & 50000 \\
        \bottomrule
    \end{tabularx}

D. Hyperparameter Importance

To analyze the importance of hyperparameter choices for $\textsc{Pet}$ 's performance gains over supervised learning, we look at the influence of both the learning rate ($\mathrm{LR}$) and the number of training steps on their test set accuracies.

We try values of ${ 1\mathrm{e}{-5}, 2\mathrm{e}{-5}, 5\mathrm{e}{-5} }$ for the learning rate and ${50, 100, 250, 500, 1000}$ for the number of training steps. As this results in 30 different configurations for just one task and training set size, we only perform this analysis on Yelp with 100 examples, for which results can be seen in Figure 6. For supervised learning, the configuration used throughout the paper ($\mathrm{LR}=1\mathrm{e}{-5}$, 250 steps) turns out to perform best whereas for $\textsc{Pet}$, training for fewer steps consistently performs even better. Importantly, $\textsc{Pet}$ clearly outperforms regular supervised training regardless of the chosen learning rate and number of training steps.

$**Figure 6:** Performance of supervised learning and $\textsc{Pet}$ (weighted, without auxiliary language modeling) for various learning rates and training steps on Yelp with 100 training examples$

E. Automatic Verbalizer Search

Given a set of patterns $P_1, \ldots, P_n$, manually finding a verbalization $v(l)$ for each $l \in \mathcal{L}$ that represents the meaning of $l$ well and corresponds to a single token in $V$ can be difficult. We therefore devise automatic verbalizer search (AVS), a procedure that automatically finds suitable verbalizers given a training set $\mathcal{T}$ and a language model $M$.

Assuming we already have a PVP $\mathbf{p} = (P, v)$, we can easily check whether some token $t \in V$ is a good verbalization of $l \in \mathcal{L}$. To this end, we define $\mathbf{p}[l\gets t] = (P, v')$, where $v'$ is identical to $v$, except that $v'(l) = t$. Intuitively, if $t$ represents $l$ well, then $q_{\mathbf{p}[l \gets t]}(l \mid \mathbf{x})$ (i.e., the probability $M$ assigns to $t$ given $P(\mathbf{x})$) should be high only for those examples $(\mathbf{x}, y) \in \mathcal{T}$ where $y = l$. We thus define the score of $t$ for $l$ given $\mathbf{p}$ as

$ \begin{split} s_{l}(t \mid \mathbf{p}) = \frac{1}{|\mathcal{T}l|} \cdot \sum{{(\mathbf{x}, y) \in \mathcal{T}l}} q{\mathbf{p}[l \gets t]}(l \mid \mathbf{x}) \ - \frac{1}{|\mathcal{T} \setminus \mathcal{T}{l}|} \cdot \sum{{(\mathbf{x}, y) \in \mathcal{T} \setminus \mathcal{T}{l}}} q{\mathbf{p}[l \gets t]}(l \mid \mathbf{x}) \end{split} $

where $\mathcal{T}_l = { (\mathbf{x}, y) \in \mathcal{T} : y = l }$ is the set of all training examples with label $l$. While this allows us to easily compute the best verbalization for $l$ as

$ \hat{t} = \arg\max_{t \in V} s_l(t \mid \mathbf{p}),, $

it requires us to already know verbalizations $v(l')$ for all other labels $l'$.

AVS solves this problem as follows: We first assign random verbalizations to all labels and then repeatedly recompute the best verbalization for each label. As we do not want the resulting verbalizer to depend strongly on the initial random assignment, we simply consider multiple such assignments. Specifically, we define an initial probability distribution $\rho_0$ where for all $t \in V, l \in \mathcal{L}$, $\rho_0(t \mid l) = 1 / |V|$ is the probability of choosing $t$ as verbalization for $l$. For each $l \in \mathcal{L}$, we then sample $k$ verbalizers $v_1, \ldots, v_k$ using $\rho_0$ to compute

$ s_l^k(t) = \frac{1}{n\cdot k} \sum_{i=1}^{n}\sum_{j=1}^k s_l(t \mid (P_i, v_j)) $

for all $t \in V$.[^6] These scores enable us to define a probability distribution $\rho_1$ that more closely reflects a word's suitability as a verbalizer for a given label:

[^6]: Note that the score $s_l^k(t)$ jointly considers all patterns; in preliminary experiments, we found this to result in more robust verbalizers.

$ \rho_1(t \mid l) = \frac{1}{Z} \max(s_l^k(t), \epsilon) $

where $Z = \sum_{t' \in V} \max(s_l^k(t'), \epsilon)$ and $\epsilon \geq 0$ ensures that $\rho_1$ is a proper probability distribution. We repeat this process to obtain a sequence of probability distributions $\rho_1, \ldots, \rho_{i_\text{max}}$. Finally, we choose the $m \in \mathbb{N}$ most likely tokens according to $\rho_{i_\text{max}}(t \mid l)$ as verbalizers for each $l$. During training and inference, we compute the unnormalized score $s_\mathbf{p}(y \mid \mathbf{x})$ for each label by averaging over its $m$ verbalizers.

We analyze the performance of AVS for all tasks with $|\mathcal{T}| = 50$ training examples and set $k = 250$, $\epsilon = 10^{-3}$, $i_\text{max} = 5$ and $m = 10$.[^7] To speed up the search, we additionally restrict our search space to tokens $t \in V$ that contain at least two alphabetic characters. Of these tokens, we only keep the $10,000$ most frequent ones in $\mathcal{D}$.

[^7]: We tried values of $k$ and $i_\text{max}$ in ${250, 500, 1000}$ and ${5, 10, 20}$, respectively, but found the resulting verbalizers to be almost identical.

Results are shown in Table 7. As can be seen, carefully handcrafted verbalizers perform much better than AVS; however, $\textsc{Pet}$ with AVS still considerably outperforms regular supervised training while eliminating the challenge of manually finding suitable verbalizers. Table 8 shows the most probable verbalizers found using AVS for the Yelp dataset. While most verbalizers for this dataset intuitively make sense, we found AVS to struggle with finding good verbalizers for three out of ten labels in the Yahoo dataset and for all MNLI labels.

\begin{tabularx}{\linewidth}{l>{\arraybackslash}X>{\arraybackslash}X>{\arraybackslash}X>{\arraybackslash}X}
        \toprule
         {} & \multicolumn{1}{c}{\textbf{Yelp}} & \multicolumn{1}{c}{\textbf{AG's}} & \multicolumn{1}{c}{\textbf{Yahoo}} & {\textbf{MNLI}} \\
        \midrule
        supervised & $44.8$ & $82.1$ & $52.5$ & $45.6$ \\
        \textsc{Pet} & $\mathbf{60.0}$ & $\mathbf{86.3}$ & $\mathbf{66.2}$ & $\mathbf{63.9}$ \\
        \textsc{Pet} + AVS & $55.2$ & $85.0$ & $58.2$ & $52.6$ \\
        \bottomrule
    \end{tabularx}

\begin{tabularx}{\linewidth}{lX}
        \toprule
        $y$ & \textbf{Top Verbalizers} \\
        \midrule
        1 & worthless, BAD, useless, appalling \\
        2 & worse, slow, frustrating, annoying\\ 
        3 & edible, mixed, cute, tasty, Okay \\
        4 & marvelous, loved, love, divine, fab \\
        5 & golden, magical, marvelous, perfection\\ 
        \bottomrule
    \end{tabularx}

References

Section Summary: The references section compiles a list of academic papers and technical reports, mostly published between 2018 and 2020, that examine large language models and their training approaches. Many entries focus on systems such as GPT, BERT, and RoBERTa, along with studies testing how well these models perform tasks like text classification, commonsense reasoning, and knowledge extraction. The works come from researchers at organizations including OpenAI and appear in venues like ACL, EMNLP, and AAAI conferences.

[1] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report.

[2] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.

[3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

[4] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. Computing Research Repository, arXiv:1907.11692.

[5] Raul Puri and Bryan Catanzaro. 2019. Zero-shot text classification with generative language models. Computing Research Repository, arXiv:1912.10165.

[6] Joe Davison, Joshua Feldman, and Alexander Rush. 2019. Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1173–1178, Hong Kong, China. Association for Computational Linguistics.

[7] Juri Opitz. 2019. Argumentative relation classification as plausibility ranking. In Preliminary proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019): Long Papers, pages 193–202, Erlangen, Germany. German Society for Computational Linguistics & Language Technology.

[8] Shashank Srivastava, Igor Labutov, and Tom Mitchell. 2018. Zero-shot learning of classifiers from natural language quantification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 306–316, Melbourne, Australia. Association for Computational Linguistics.

[9] Zied Bouraoui, Jose Camacho-Collados, and Steven Schockaert. 2020. Inducing relational knowledge from BERT. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence.

[10] Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. Computing Research Repository, arXiv:1806.08730.

[11] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.

[12] Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. Computing Research Repository, arXiv:1806.02847.

[13] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

[14] Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019. Does it make sense? And why? A pilot study for sense making and explanation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4020–4026, Florence, Italy. Association for Computational Linguistics.

[15] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. WinoGrande: An adversarial winograd schema challenge at scale. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence.

[16] Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48.

[17] Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811–7818, Online. Association for Computational Linguistics.

[18] Timo Schick and Hinrich Schütze. 2020. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence.

[19] Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2019. oLMpics – on what language model pre-training captures. Computing Research Repository, arXiv:1912.13283.

[20] Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438.

[21] Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. 2018. Diverse few-shot text classification with multiple metrics. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1206–1215, New Orleans, Louisiana. Association for Computational Linguistics.

[22] Jiatao Gu, Yong Wang, Yun Chen, Victor O. K. Li, and Kyunghyun Cho. 2018. Meta-learning for low-resource neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3622–3631, Brussels, Belgium. Association for Computational Linguistics.

[23] Zi-Yi Dou, Keyi Yu, and Antonios Anastasopoulos. 2019. Investigating meta-learning algorithms for low-resource natural language understanding tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1192–1197, Hong Kong, China. Association for Computational Linguistics.

[24] Kun Qian and Zhou Yu. 2019. Domain adaptive dialog generation via meta learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2639–2649, Florence, Italy. Association for Computational Linguistics.

[25] Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3914–3923, Hong Kong, China. Association for Computational Linguistics.

[26] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. 2020. Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Systems, volume 33. Curran Associates, Inc.

[27] Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. MixText: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2147–2157, Online. Association for Computational Linguistics.

[28] Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational Linguistics.

[29] Bernardino Romera-Paredes and Philip Torr. 2015. An embarrassingly simple approach to zero-shot learning. In International Conference on Machine Learning, pages 2152–2161.

[30] Sappadla Prateek Veeranna, Jinseok Nam, Eneldo Loza Mencıa, and Johannes Fürnkranz. 2016. Using semantic similarity for multi-label zero-shot classification of text documents. In Proceeding of European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. Bruges, Belgium: Elsevier, pages 423–428.

[31] Zhiquan Ye, Yuxia Geng, Jiaoyan Chen, Jingmin Chen, Xiaoxiao Xu, SuHang Zheng, Feng Wang, Jun Zhang, and Huajun Chen. 2020. Zero-shot text classification via reinforced self-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3014–3024, Online. Association for Computational Linguistics.

[32] David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd Annual Meeting of the Association for Computational Linguistics, pages 189–196, Cambridge, Massachusetts, USA. Association for Computational Linguistics.

[33] Sergey Brin. 1999. Extracting patterns and relations from the world wide web. In The World Wide Web and Databases, pages 172–183, Berlin, Heidelberg. Springer Berlin Heidelberg.

[34] Eugene Agichtein and Luis Gravano. 2000. Snowball: Extracting relations from large plain-text collections. In Proceedings of the Fifth ACM Conference on Digital Libraries, DL ’00, page 85–94, New York, NY, USA. Association for Computing Machinery.

[35] David S. Batista, Bruno Martins, and Mário J. Silva. 2015. Semi-supervised bootstrapping of relationship extractors with distributional semantics. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 499–504, Lisbon, Portugal. Association for Computational Linguistics.

[36] David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 152–159, New York City, USA. Association for Computational Linguistics.

[37] Roi Reichart and Ari Rappoport. 2007. Self-training for enhancement and domain adaptation of statistical parsers trained on small datasets. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 616–623, Prague, Czech Republic. Association for Computational Linguistics.

[38] Zhongqiang Huang and Mary Harper. 2009. Self-training PCFG grammars with latent annotations across languages. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 832–841, Singapore. Association for Computational Linguistics.

[39] Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. 2018. Iterative back-translation for neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 18–24, Melbourne, Australia. Association for Computational Linguistics.

[40] Junxian He, Jiatao Gu, Jiajun Shen, and Marc'Aurelio Ranzato. 2020. Revisiting self-training for neural sequence generation. In International Conference on Learning Representations.

[41] Alexandra Chronopoulou, Christos Baziotis, and Alexandros Potamianos. 2019. An embarrassingly simple approach for transfer learning from pretrained language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2089–2095, Minneapolis, Minnesota. Association for Computational Linguistics.

[42] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. Computing Research Repository, arXiv:1503.02531.

[43] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 1321–1330. JMLR.org.

[44] Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 649–657. Curran Associates, Inc.

[45] Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics.

[46] Jannis Vamvas and Rico Sennrich. 2020. X-stance: A multilingual multi-target dataset for stance detection. Computing Research Repository, arXiv:2003.08385.

[47] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.

[48] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32, pages 5753–5763. Curran Associates, Inc.

[49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc.

[50] Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics.

[51] Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune BERT for text classification? In Chinese Computational Linguistics, pages 194–206, Cham. Springer International Publishing.

[52] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.

[53] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop.