Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

Sewon MinXinxi LyuAri HoltzmanMikel ArtetxeMike LewisHannaneh HajishirziLuke Zettlemoyer

article2022EMNLP2,180 citations

Reveals that large language models do not rely on correct demonstration labels for in-context learning, showing that performance is primarily driven by prompt format, input distribution, and label space rather than input-label mappings.

Listen

Large language models have shown remarkable capabilities in performing new downstream tasks purely through in-context learning—conditioning on a few prompt demonstrations without fine-tuning model parameters. Conventionally, practitioners assumed that this mechanism relies on the model learning the true relationship between inputs and correct outputs from the provided examples. However, data annotation is costly and prone to errors, making it critical to understand what elements of these prompt demonstrations genuinely drive task performance and whether accurate labels are required.

The article evaluates which components of demonstrations contribute to in-context learning success and directly tests whether large language models require ground-truth input-label mappings to perform tasks effectively.

To investigate this, the researchers conducted systematic empirical evaluations across 12 dense language model configurations, ranging in size from 774 million to 175 billion parameters, including GPT-2, GPT-J, fairseq models, MetaICL, and GPT-3. The study evaluated performance across 26 low-resource datasets spanning sentiment analysis, paraphrase detection, natural language inference, hate speech detection, question answering, and sentence completion. By systematically altering demonstration inputs, label validity, label vocabularies, and overall format structures, the authors isolated the individual impact of four core aspects: the input-label mapping, the input text distribution, the label space, and the sequence format.

The findings reveal that ground-truth labels matter surprisingly little: replacing true demonstration labels with random labels caused only a marginal drop in performance, typically between 0% and 5% absolute across evaluated models (averaging a 1.7% drop in multi-choice tasks and 2.6% in classification). In fact, prompts with 100% incorrect labels still achieved the vast majority of performance gains over standard zero-shot baselines. Instead of input-label pairings, task performance is primarily driven by three structural factors: providing examples of the expected input distribution, exposing the valid label space, and maintaining the sequence format of input-label pairs. In models meta-trained specifically for in-context learning, this effect is amplified, as the models rely almost entirely on formatting and label-space cues rather than the underlying input-label correspondences.

These insights demonstrate that language models do not learn entirely new task mappings at inference time; rather, demonstrations serve to activate and locate capabilities, task structures, and concepts already acquired during pre-training. For organizations deploying language models, these findings offer substantial operational value by reducing data labeling costs, mitigating risks tied to noisy annotations, and establishing that strong zero-shot or few-shot performance can often be achieved using unlabeled text paired with randomized labels.

Organizations should focus prompt-engineering investments on presenting realistic input distributions, specifying well-defined candidate label spaces, and maintaining consistent prompt formatting rather than investing heavily in exhaustive data labeling for in-context tasks. Researchers and practitioners should validate whether custom downstream tasks rely on concepts existing in the pre-trained model; when entirely new task semantics are required, direct model fine-tuning remains necessary.

Confidence in these conclusions is high for standard classification and multiple-choice NLP benchmarks across diverse model families and scales. However, caution is warranted when extrapolating to open-ended text generation, multi-step mathematical reasoning, or highly synthetic tasks where precise reasoning chains or unique mappings may play a larger role. Further evaluation is recommended before extending these conclusions beyond discriminative tasks.

Allen Institute for AI Meta University of Washington classification demonstrations inference Demonstrations Labels Learning

arXiv: 2202.12837 Alrope123/rethinking-demonstrations

Paper: Language Models are Few-Shot Learners, T. B. Brown et al. (2020). Reading GPT-3's introduction provides the exact few-shot in-context learning baseline and model capabilities that the source paper investigates and challenges.
Paper: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, Jason Wei et al. (2022). Understanding chain-of-thought prompting offers crucial background on how demonstrations guide large language models through reasoning steps.

Paper: Finetuned Language Models Are Zero-Shot Learners, Jason Wei et al. (2022). This paper builds on the source's findings about prompt formatting by demonstrating how instruction tuning enables zero-shot generalization across unseen tasks.
Paper: Learning by Distilling Context, Charlie Snell et al. (2022). This work directly extends the source by showing how demonstration context can be permanently distilled into model parameters rather than supplied at inference time.
Paper: Making Text Embedders Few-Shot Learners, Chaofan Li et al. (2025). This work applies the source's insights on in-context learning to text embeddings, showing how few-shot examples improve representation quality.

Cover for Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

Abstract

Large language models (LMs) are able to in-context learn -- perform a new task via inference alone by conditioning on a few input-label pairs (demonstrations) and making predictions for new inputs. However, there has been little understanding of how the model learns and which aspects of the demonstrations contribute to end task performance. In this paper, we show that ground truth demonstrations are in fact not required -- randomly replacing labels in the demonstrations barely hurts performance on a range of classification and multi-choce tasks, consistently over 12 different models including GPT-3. Instead, we find that other aspects of the demonstrations are the key drivers of end task performance, including the fact that they provide a few examples of (1) the label space, (2) the distribution of the input text, and (3) the overall format of the sequence. Together, our analysis provides a new way of understanding how and why in-context learning works, while opening up new questions about how much can be learned from large language models through inference alone.

Topics

classification demonstrations inference language model large language models lms

Keywords

Demonstrations Labels Learning In-Context Learning Random Labels Channel Direct MetaICL Models Label Space

1 Introduction
2 Related Work
3 Experimental Setup
4 Ground Truth Matters Little
4.1 Gold labels vs. random labels
4.2 Ablations
5 Why does In-Context Learning work?
5.1 Impact of the distribution of the input text
5.2 Impact of the label space
5.3 Impact of input-label pairing
5.4 Impact of meta-training
6 Discussion & Conclusion
References
A Full Datasets
B Experimental Details
C More Experimental Results
C.1 Gold labels vs. random labels
C.2 Random labels from true distribution of labels & Task breakdown
C.3 More variants of the demonstrations

Knowls

Knowl 1 — Robustness of In-Context Learning to Random Demonstration Labels
empirical result
Evaluating autoregressive language models on text classification and multi-choice benchmarks demonstrates that replacing ground-truth labels in few-shot demonstrations with labels sampled uniformly at random from the candidate label space results in only a marginal decrease in downstream performance (typically 0% to 5% absolute). Across 12 dense decoder-only models ranging from 774M to 175B parameters—including GPT-2 Large (774M), MetaICL (774M), GPT-J (6B), fairseq LMs (6.7B and 13B), and GPT-3 (175B Davinci base)—and across 26 low-resource datasets using $k=16$ demonstrations, the average performance drop from gold to random labels is 2.6% absolute Macro-F1 on classification tasks and 1.7% absolute Accuracy on multi-choice tasks.

Both gold-label demonstrations and random-label demonstrations substantially outperform zero-shot inference (no demonstrations). This indicates that in-context learning does not primarily rely on the model learning input-label correspondences at test time.
Knowl 2 — Four Structural and Semantic Components of In-Context Demonstrations
definition
In-context learning prompts consisting of $k$ demonstration pairs $(x_1, y_1), (x_2, y_2), \dots, (x_k, y_k)$ and a test input $x$ provide four distinct learning signals to a language model:
1. Input-label mapping: The correctness of the paired correspondence between each input $x_i$ and its ground-truth target label $y_i$ .
2. Input text distribution: The underlying linguistic and domain distribution from which the demonstration inputs $x_1, \dots, x_k$ are drawn (e.g., domain-specific task sentences versus out-of-distribution natural text).
3. Label space: The discrete set of valid target categories or candidate tokens $C = \{y_1, \dots, y_k\}$ defining the range of expected model outputs.
4. Demonstration format: The sequence syntax and structural template conventions used to present input-label pairs, specifically the alternating presentation and delimiter structure (e.g., input followed by delimiter and label, separated by consistent newline tokens).
Knowl 3 — Impact of In-Distribution Input Text on In-Context Performance
empirical result
Replacing demonstration inputs $x_1, \dots, x_k$ with out-of-distribution (OOD) sentences sampled from an external corpus (such as CC-News) while preserving the target label space $C$ and the input-label pairing format leads to substantial performance drops. Across benchmark evaluations using Channel MetaICL, Direct GPT-J, and Channel GPT-J on classification and multi-choice tasks, replacing in-distribution inputs with OOD inputs reduces performance by 3% to 16% absolute in Macro-F1 / Accuracy.

In some direct prompting configurations, such as Direct GPT-J on multi-choice datasets, conditioning on OOD demonstration inputs results in performance significantly worse than zero-shot inference without demonstrations. This confirms that conditioning on in-distribution input text is a primary driver of few-shot prompting performance, making the prompt sequence closer to the pretraining distribution.
Knowl 4 — Asymmetric Dependence of Direct and Channel Prompting on the Demonstration Label Space
empirical result
The necessity of providing demonstrations within the valid task label space depends on the inference formulation used:
- Direct Prompting (modeling $P(y \mid x_1, y_1, \dots, x_k, y_k, x)$ ): Replacing demonstration labels with randomly sampled English words from outside the task's label space ( $y_i \in C_{\text{rand}}$ where $|C_{\text{rand}}| = |C|$ ) results in severe performance degradation of 5% to 16% absolute Macro-F1 / Accuracy. Direct models rely heavily on the demonstrations to specify the discrete set of candidate label words to generate.
- Channel Prompting (modeling $P(x \mid x_1, y_1, \dots, x_k, y_k, y)$ ): Replacing demonstration labels with random English words causes negligible performance changes (between 0% and 2% absolute drop, and occasionally slight gains). Because channel models condition on candidate labels to estimate the probability of generating the input text rather than generating label tokens directly, they do not depend on the demonstrations to delimit the label space.
Knowl 5 — Role of Sequence Formatting in Retaining In-Context Learning Gains
empirical result
Maintaining the structured format of alternating input-label pairs is essential for few-shot prompt performance. Completely removing the input-label pairing format—such as conditioning the model on concatenated inputs alone ( $x_1 \dots x_k$ ) or concatenated labels alone ( $y_1 \dots y_k$ )—yields performance that is comparable to or worse than zero-shot inference without demonstrations.

Conversely, retaining the delimiter and pairing format while stripping other components allows models to preserve most few-shot gains:
- Direct MetaICL achieves 95% (classification) and 82% (multi-choice) of the gain over zero-shot by pairing out-of-distribution text with valid labels.
- Channel MetaICL and Channel GPT-J preserve between 75% and 87% of the few-shot gain by pairing in-distribution inputs with arbitrary English words.
Replacing demonstration labels with a single constant token (such as "answer") or replicating the test input across all demonstration slots degrades performance, as constant tokens disrupt the perceived delimiter structure between examples.
Knowl 6 — Insensitivity of In-Context Performance to Demonstration Label Accuracy Proportions
empirical result
Varying the proportion of correct demonstration labels $a \in [0, 100]\%$ demonstrates that language model performance is largely invariant to label accuracy. When evaluating classification and multi-choice benchmarks with $k=16$ demonstrations:
- Completely corrupted demonstrations ( $a = 0\%$ , where every input $x_i$ is explicitly paired with an incorrect label $y \in C \setminus \{y_i\}$ ) preserve 92% of the few-shot gain over zero-shot in Channel MetaICL classification, 100% in Channel MetaICL multi-choice, and 97% in Direct GPT-J multi-choice.
- Across models and tasks, demonstrations with 0% correct labels consistently and significantly outperform zero-shot inference with no demonstrations.
Knowl 7 — Demonstration Construction with Controlled Label Accuracy
algorithm
The following procedure constructs a demonstration set of $k$ input-label pairs with an exact label accuracy percentage $a \in [0, 100]\%$ , enabling the controlled evaluation of label correctness in in-context learning:
Input: Labeled training examples $S = \{(x_i, y_i)\}_{i=1}^k$ , label set $C$ , target accuracy percentage $a \in [0, 100]$
Output: Formed demonstration list $D$
procedure FormDemons( $S$ , $a$ )
     $D \leftarrow []$
     $n \leftarrow \lfloor k \times a / 100 \rfloor$
     $G \leftarrow \text{Sample}(\{1, 2, \dots, k\}, n)$
    for $i = 1$ to $k$ do
        if $i \in G$ then
             $D.\text{append}((x_i, y_i))$
        else
             $y_{\text{rand}} \leftarrow \text{Sample}(C \setminus \{y_i\})$
             $D.\text{append}((x_i, y_{\text{rand}}))$
    return $D$
The algorithm uniformly samples $n$ indices to retain their gold ground-truth labels $y_i$ . For the remaining $k - n$ indices, it replaces $y_i$ with an incorrect label sampled uniformly at random from the candidate label space excluding the gold label, $C \setminus \{y_i\}$ .
Knowl 8 — Saturation of In-Context Learning Performance with Demonstration Count
empirical result
Evaluating in-context performance across demonstration counts $k \in \{0, 4, 8, 16, 32\}$ shows that accuracy increases sharply from $k=0$ (zero-shot) to $k=4$ , but plateaus for $k \ge 8$ . This saturation behavior occurs identically for demonstrations with ground-truth labels and demonstrations with random labels, with the performance gap between gold and random labels remaining consistently between 0.8% and 1.6% across varying $k$ .

This behavior contrasts with standard supervised training, where increasing labeled training data continuously improves performance by refining the learned input-label mapping. In in-context learning, the primary drivers of performance (input distribution, label space, and format structure) are recovered from very few examples ( $k \approx 4\text{--}8$ ).
Knowl 9 — Meta-Training Amplifies Demonstration Format Exploitation Over Input-Label Mapping
empirical result
Meta-training a language model on diverse supervised datasets using an in-context learning objective (as implemented in MetaICL, initialized from GPT-2 Large 774M) magnifies the model's reliance on superficial demonstration characteristics while reducing reliance on true input-label mappings:
- The performance drop when replacing gold labels with uniformly random labels is only 0.1% to 0.9% absolute in MetaICL, compared to up to 5.0% in non-meta-trained models such as GPT-J.
- In Direct MetaICL, the input distribution has near-zero impact (pairing out-of-distribution sentences with valid label sets retains 95% of the few-shot gain over zero-shot).
- In Channel MetaICL, the label space has near-zero impact (pairing in-distribution inputs with arbitrary English words retains 82% of the few-shot gain over zero-shot).
Meta-training trains the model to exploit easily extractable prompt cues (format syntax and target generation space) rather than acquiring task-specific input-label mappings at test time.
Knowl 10 — Direct and Channel In-Context Prompting Formulations
model/method
For a classification or multi-choice task with discrete label set $C$ , input $x$ , and $k$ demonstration pairs $(x_1, y_1), \dots, (x_k, y_k)$ , test predictions are computed via two inference formulations:
1. Direct In-Context Prompting: Computes the conditional likelihood of generating candidate label $y \in C$ given the concatenated demonstrations and the query input $x$ : $y^* = \arg\max_{y \in C} P(y \mid x_1, y_1, \dots, x_k, y_k, x)$
2. Channel In-Context Prompting: Inverts the conditioning direction using Bayes' rule, computing the conditional likelihood of generating query input $x$ given candidate label $y \in C$ : $y^* = \arg\max_{y \in C} P(x \mid y_1, x_1, \dots, y_k, x_k, y)$
In zero-shot settings ( $k=0$ ), direct inference computes $\arg\max_{y \in C} P(y \mid x)$ and channel inference computes $\arg\max_{y \in C} P(x \mid y)$ .

Coverage note — None omitted; all primary empirical discoveries, prompt component factorizations, control algorithms, and analytical conclusions are covered.

References

1.Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with pre-finetuning. arXiv preprint arXiv:2101.11038.
2.Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, et al. 2021. Efficient large scale language modeling with mixtures of experts. arXiv preprint arXiv:2112.10684.
3.Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment.
4.Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of EMNLP.
5.Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. In TAC.
6.Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NeurIPS.
7.Michael Chen, Mike D’Arcy, Alisa Liu, Jared Fernandez, and Doug Downey. 2019. CODAH: An adversarially-authored question answering dataset for common sense. In Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP.
8.Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. 2021. Meta-learning via language model in-context tuning. arXiv preprint arXiv:2110.07814.
9.Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv.
10.Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop.
11.Ona de Gibert, Naiara Perez, Aitor García-Pablos, and Montse Cuadros. 2018. Hate Speech Dataset from a White Supremacy Forum. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2).
12.Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitmentbank: Investigating projection in naturally occurring discourse. Proceedings of Sinn und Bedeutung.
13.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL.
14.T. Diggelmann, Jordan L. Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2020. Climate-fever: A dataset for verification of real-world climate claims. ArXiv.
15.William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
16.Avia Efrat and Omer Levy. 2020. The turking test: Can language models understand instructions? arXiv preprint arXiv:2010.11982.
17.L Gao, S Biderman, S Black, L Golding, T Hoppe, C Foster, J Phang, H He, A Thite, N Nabeshima, et al. 2021. The pile: an 800gb dataset of diverse text for language modeling 2020. arXiv preprint arXiv:2101.00027.
18.Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing.
19.Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In The First Joint Conference on Lexical and Computational Semantics (SemEval).
20.Ari Holtzman, Peter West, Vered Schwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competition: Why the highest probability answer isn’t always right. In EMNLP.
21.Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UnifiedQA: Crossing format boundaries with a single qa system. In Findings of EMNLP.
22.Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. In AAAI.
23.Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, Kang Min Yoo, and Taeuk Kim. 2022. Ground-truth labels matter: A deeper look into input-label demonstrations. arXiv preprint arXiv:2205.12685.
24.Elyor Kodirov, Tao Xiang, Zhenyong Fu, and Shaogang Gong. 2015. Unsupervised domain adaptation for zero-shot learning. In Proceedings of the IEEE international conference on computer vision.
25.Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning.
26.Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL.
27.Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In EMNLP: System Demonstrations.
28.Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804.
29.Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
30.Robert L Logan IV, Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2021. Cutting down on prompts and parameters: Simple few-shot learning with language models. arXiv preprint arXiv:2106.13353.
31.Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786.
32.Aman Madaan and Amir Yazdanbakhsh. 2022. Text and patterns: For effective chain of thought, it takes two to tango. arXiv preprint arXiv:2209.07686.
33.Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala. 2014. Good debt or bad debt: Detecting semantic orientations in economic texts. J. Assoc. Inf. Sci. Technol.
34.Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In LREC.
35.Clara H. McCreery, Namit Katariya, Anitha Kannan, Manish Chablani, and Xavier Amatriain. 2020. Effective transfer learning for identifying similar questions: Matching user questions to covid-19 faqs. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.
36.Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP.
37.Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2021a. Noisy channel language model prompting for few-shot text classification. arXiv preprint arXiv:2108.04106.
38.Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2021b. MetaICL: Learning to learn in context. arXiv preprint.
39.Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. 2021a. Reframing instructional prompts to gptk’s language. arXiv preprint arXiv:2109.07830.
40.Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021b. Cross-task generalization via natural language crowdsourcing instructions. arXiv preprint arXiv:2104.08773.
41.Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. 2020. Ethos: an online hate speech detection dataset. ArXiv.
42.Sebastian Nagel. 2016. CC-News. http://web.archive.org/save/http://commoncrawl.org/2016/10/news-dataset-available.
43.Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog.
44.Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research.
45.Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot reasoning. arXiv preprint arXiv:2202.07206.
46.Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems.
47.Frieda Rong. 2021. Extrapolating to unnatural language processing with gpt-3’s in-context learning: The good, the bad, and the mysterious. https://ai.stanford.edu/blog/in-context-learning.
48.Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.08633.
49.Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2022. Multitask prompted training enables zero-shot task generalization. In ICLR.
50.Timo Schick and Hinrich Schütze. 2021. It’s not just size that matters: Small language models are also few-shot learners. In NAACL-HLT.
51.Emily Sheng and David Uthus. 2020. Investigating societal biases in a poetry composition system. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing.
52.Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. DREAM: A challenge data set and models for dialogue-based reading comprehension. TACL.
53.Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. 2019a. Quarel: A dataset and models for answering questions about qualitative relationships. In AAAI.
54.Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019b. QuaRTz: An open-domain dataset of qualitative relationship questions. In EMNLP.
55.Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In NAACL-HLT.
56.Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS.
57.Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP.
58.Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax.
59.Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao. 2019b. A survey of zero-shot learning: Settings, methods, and applications. ACM Transactions on Intelligent Systems and Technology (TIST).
60.Albert Webson and Ellie Pavlick. 2022. Do prompt-based models really understand the meaning of their prompts? In NAACL-HLT.
61.Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2022a. Finetuned language models are zero-shot learners. In ICLR.
62.Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022b. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903.
63.Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022. An explanation of in-context learning as implicit bayesian inference. In ICLR.
64.Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. Crossfit: A few-shot learning challenge for cross-task generalization in nlp. In EMNLP.
65.Tony Z Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In ICML.

Citation

MLA

Min, S., et al. “Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?”. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 11048–64, https://doi.org/10.18653/v1/2022.emnlp-main.759.

APA

Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022). Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 11048–11064. https://doi.org/10.18653/v1/2022.emnlp-main.759

Chicago

Min, S., X. Lyu, A. Holtzman, et al. 2022. “Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?”. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 11048–64. https://doi.org/10.18653/v1/2022.emnlp-main.759.

Harvard

Min, S. et al. (2022) “Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?”, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp. 11048–11064. Available at: https://doi.org/10.18653/v1/2022.emnlp-main.759.

Vancouver

1. Min S, Lyu X, Holtzman A, Artetxe M, Lewis M, Hajishirzi H, Zettlemoyer L (2022) Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp 11048–11064

BibTeX

@inproceedings{min-etal-2022-rethinking,
    title = "Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?",
    author = "Min, Sewon  and
      Lyu, Xinxi  and
      Holtzman, Ari  and
      Artetxe, Mikel  and
      Lewis, Mike  and
      Hajishirzi, Hannaneh  and
      Zettlemoyer, Luke",
    editor = "Goldberg, Yoav  and
      Kozareva, Zornitsa  and
      Zhang, Yue",
    booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.emnlp-main.759/",
    doi = "10.18653/v1/2022.emnlp-main.759",
    pages = "11048--11064"
}

Metadata:ACL Anthology

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Robustness of In-Context Learning to Random Demonstration Labels

Knowl 2 — Four Structural and Semantic Components of In-Context Demonstrations

Knowl 3 — Impact of In-Distribution Input Text on In-Context Performance

Knowl 4 — Asymmetric Dependence of Direct and Channel Prompting on the Demonstration Label Space

Knowl 5 — Role of Sequence Formatting in Retaining In-Context Learning Gains

Knowl 6 — Insensitivity of In-Context Performance to Demonstration Label Accuracy Proportions

Knowl 7 — Demonstration Construction with Controlled Label Accuracy

Knowl 8 — Saturation of In-Context Learning Performance with Demonstration Count

Knowl 9 — Meta-Training Amplifies Demonstration Format Exploitation Over Input-Label Mapping

Knowl 10 — Direct and Channel In-Context Prompting Formulations

References

Citation

BibTeX

Source Code

Access the Paper