What Makes Good In-Context Examples for GPT-3?

Jiachang LiuDinghan ShenYizhe ZhangBill DolanLawrence CarinWeizhu Chen

article2021Workshop on Knowledge Extraction and Integration for Deep Learning Architectures; Deep Learning Inside Out1,881 citations

Proposes a retrieval-based method to select semantically similar in-context examples for GPT-3, demonstrating that similarity-driven prompt selection dramatically outperforms random sampling across diverse generation and question-answering benchmarks.

Listen

Large language models such as GPT-3 have demonstrated impressive capabilities in learning tasks directly from a few prompt examples without requiring model fine-tuning. However, standard deployments typically select these demonstration examples at random, leading to significant performance instability and unpredictable accuracy across different inputs. Because fine-tuning massive models requires prohibitive computational infrastructure and parameter access is often restricted, finding practical, lightweight methods to stabilize and maximize prompt performance is a critical priority for practical deployment.

The article evaluates whether systematically retrieving demonstration examples that are semantically similar to a given query can significantly enhance GPT-3's task performance and reliability. To achieve this, the authors introduce KATE, a non-parametric method that uses separate sentence-embedding models to identify and rank nearest-neighbor examples from available training pools to form the context prompt for each input query.

The approach was evaluated across diverse natural language processing benchmarks, including sentiment analysis on IMDB, structured table-to-text generation on the ToTTo benchmark, and open-domain question answering on the Natural Questions, Web Questions, and TriviaQA datasets. The authors assessed performance against standard random sampling, direct nearest-neighbor baselines without language models, and fully fine-tuned baseline models such as T5.

The key findings reveal substantial, consistent performance gains from semantic example selection. First, retrieving semantically closest neighbors dramatically outperformed random selection and farthest-neighbor baselines across all tasks, boosting exact-match question-answering accuracy on Natural Questions from 28.6% up to 41.6% and table-to-text generation scores from 28.4 to 41.0 BLEU points. Second, augmenting GPT-3 with this retrieval method enabled it to match or exceed the accuracy of dedicated, fully fine-tuned T5 models on complex generation and question-answering benchmarks. Third, qualitative analysis showed that semantically aligned examples substantially reduced factual hallucinations in text generation by providing relevant structural templates. Finally, performance scaled positively with larger example pools and larger prompt context windows, while showing low sensitivity to example ordering.

These results demonstrate that large language models act far more effectively as universal problem solvers when guided by relevant context, circumventing the high financial and storage costs of maintaining specialized fine-tuned checkpoints for distinct tasks. Organizations deploying prompt-based language models should implement nearest-neighbor retrieval pipelines to assemble dynamic prompt contexts, ensuring the embedding retrieval models are aligned with the target domain. While the method delivers strong gains, leaders should note that the approach relies on having accessible labeled candidate data pools and requires managing API inference costs when utilizing larger prompt windows.

Duke University Meta Microsoft encoders few-shot learning genetic selection Examples In-Context Examples GPT-3

arXiv: 2101.06804

Paper: Language Models are Few-Shot Learners, T. B. Brown et al. (2020). Introduces the GPT-3 foundation model and formalizes the few-shot in-context learning paradigm that this source analyzes and improves upon.
Paper: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Patrick Lewis et al. (2020). Establishes the retrieval-augmented framework for language models that directly inspires using nearest-neighbor retrieval to select prompt examples.
Paper: Calibrate Before Use: Improving Few-Shot Performance of Language Models, Tony Z. Zhao et al. (2021). Identifies the severe sensitivity and instability of GPT-3 to in-context prompt example selection and ordering, motivating systematic selection strategies.
Paper: Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference, Timo Schick et al. (2020). Demonstrates the efficacy of cloze-style prompting and pattern exploitation to elicit latent task knowledge from language models in low-resource settings.
Paper: Making Pre-trained Language Models Better Few-shot Learners, Tianyu Gao et al. (2021). Pioneers selective semantic demonstration sampling for prompt-based few-shot learners using sentence embeddings.

Paper: Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?, Sewon Min et al. (2022). Investigates what specific mechanisms make in-context demonstrations effective, challenging assumptions about ground-truth input-label mappings.
Paper: REPLUG: Retrieval-Augmented Black-Box Language Models, Weijia Shi et al. (2024). Extends retrieval augmentation to black-box large language models through parallel document prompting and retriever tuning.
Paper: In-Context Retrieval-Augmented Language Models, Ori Ram et al. (2023). Applies non-parametric retrieval directly into the context window of standard, unmodified language models across varied scales.
Paper: Retrieval-Augmented Generation for Large Language Models: A Survey, Yunfan Gao et al. (2023). Provides a comprehensive taxonomy and evaluation of retrieval-augmented generation paradigms built upon dynamic context selection.
Paper: Lost in the Middle: How Language Models Use Long Contexts, Nelson F. Liu et al. (2024). Examines how language models utilize information across long context windows, diagnosing positional biases in retrieved context.
Paper: Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, Akari Asai et al. (2024). Develops a framework for models to dynamically retrieve, generate, and self-critique contextual passages during inference.
Paper: Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning, Haokun Liu et al. (2022). Critiques the sample and compute efficiency of in-context few-shot learning by comparing it against lightweight parameter-efficient fine-tuning.

Cover for What Makes Good In-Context Examples for GPT-3?

Abstract

GPT- $3$ has attracted lots of attention due to its superior performance across a wide range of NLP tasks, especially with its powerful and versatile in-context few-shot learning ability. Despite its success, we found that the empirical results of GPT- $3$ depend heavily on the choice of in-context examples. In this work, we investigate whether there are more effective strategies for judiciously selecting in-context examples (relative to random sampling) that better leverage GPT- $3$ 's few-shot capabilities. Inspired by the recent success of leveraging a retrieval module to augment large-scale neural network models, we propose to retrieve examples that are semantically-similar to a test sample to formulate its corresponding prompt. Intuitively, the in-context examples selected with such a strategy may serve as more informative inputs to unleash GPT- $3$ 's extensive knowledge. We evaluate the proposed approach on several natural language understanding and generation benchmarks, where the retrieval-based prompt selection approach consistently outperforms the random baseline. Moreover, it is observed that the sentence encoders fine-tuned on task-related datasets yield even more helpful retrieval results. Notably, significant gains are observed on tasks such as table-to-text generation (41.9% on the ToTTo dataset) and open-domain question answering (45.5% on the NQ dataset). We hope our investigation could help understand the behaviors of GPT- $3$ and large-scale pre-trained LMs in general and enhance their few-shot capabilities.

1 Introduction
2 Method
2.1 GPT-33 for In-Context Learning
2.2 The Impact of In-Context Examples
2.3 kkNN-augmented In-Context Example Selection
3 Experimental Setup
3.1 Sentence Embeddings for Retrieval
3.2 Baseline Methods
4 Experimental Results
4.1 Sentiment Analysis
4.2 Table-to-text Generation
4.3 Questing Answering
5 Analysis and Ablation Study
5.1 Number of In-context Examples
5.2 Size of Training Set for Retrieval
5.3 Order of In-context Examples
6 Related Work
7 Conclusion
References

Knowls

Knowl 1 — KATE: kNN-Augmented In-Context Example Selection
algorithm
KATE ( $k\text{NN}$ -Augmented in-conText Example selection) is a non-parametric method to select demonstration examples for in-context learning in large language models such as GPT-3 without updating model parameters. Given a test input, KATE retrieves the $k$ nearest training examples in the embedding space of a neural sentence encoder and constructs an ordered prompt prefix.
Input: Test query $x_{\text{test}}$ , labeled training datastore $D_T = \{(x_i, y_i)\}_{i=1}^N$ , sentence encoder $\mu_\theta(\cdot)$ , distance/similarity function $s(\cdot, \cdot)$ , number of in-context examples $k$
Output: Predicted output $\hat{y}_{\text{test}}$
$v_{\text{test}} \leftarrow \mu_\theta(x_{\text{test}})$
for $i = 1$ to $N$ do
$v_i \leftarrow \mu_\theta(x_i)$
$s_i \leftarrow s(v_{\text{test}}, v_i)$ // e.g., negative Euclidean distance $-\|v_{\text{test}} - v_i\|_2$ or cosine similarity
end for
$\{\sigma(1), \sigma(2), \dots, \sigma(k)\} \leftarrow$ indices of the $k$ largest similarities in descending order such that $s_{\sigma(1)} \ge s_{\sigma(2)} \ge \dots \ge s_{\sigma(k)}$
$C \leftarrow [x_{\sigma(1)}; y_{\sigma(1)}; x_{\sigma(2)}; y_{\sigma(2)}; \dots; x_{\sigma(k)}; y_{\sigma(k)}]$
$\hat{y}_{\text{test}} \leftarrow \text{GPT-3}([C; x_{\text{test}}])$
return $\hat{y}_{\text{test}}$
The prompt is formulated by concatenating retrieved input-output pairs with a delimiter string (such as \n), followed by the test input $x_{\text{test}}$ . The language model generates tokens autoregressively conditioned on $C$ and $x_{\text{test}}$ using greedy decoding (temperature $T=0$ ).
Knowl 2 — Conditional Autoregressive Formulation of In-Context Generation
equation
In-context learning with an autoregressive language model parameterised by $\text{LM}$ is formulated as conditional sequence generation. The probability of generating a target sequence $y = (y_1, y_2, \dots, y_T)$ given a source input $x$ and an in-context demonstration prompt $C$ consisting of $k$ paired examples is:

$p_{\text{LM}}(y \mid C, x) = \prod_{t=1}^T p(y_t \mid C, x, y_{<t})$

where $C = (x_1, y_1, x_2, y_2, \dots, x_k, y_k)$ is a single concatenated context string formatted with line break separators \n, $y_{<t} = (y_1, \dots, y_{t-1})$ denotes the sequence of preceding target tokens generated prior to time step $t$ , and $T$ is the total token length of target sequence $y$ .
Knowl 3 — Correlation Between Prompt Semantic Distance and In-Context Generation Accuracy
empirical result
The performance of GPT-3 in-context learning is strongly correlated with the semantic proximity between the demonstration examples and the test query in sentence embedding space.

On an evaluation subset of 100 questions from the Natural Questions (NQ) dataset using the [CLS] token embeddings of a pretrained RoBERTa-large model and Euclidean distance:
- Using the 10 closest neighbors from the training pool as in-context prompts achieved an Exact Match (EM) accuracy of 46.0%.
- Using the 10 farthest instances from the training pool as in-context prompts achieved an Exact Match (EM) accuracy of 31.0%.
This demonstrates that selecting semantically close training instances provides substantially higher prompt utility than selecting distant instances.

Knowl 4 — Cross-Dataset Sentiment Analysis Evaluation Under KATE

data/table

The performance of GPT-3 on binary sentiment analysis was evaluated on the IMDB test set using $k=3$ in-context demonstration examples selected from the SST-2 training set. This cross-dataset transfer setup evaluates demonstration selection across different domain distributions.

Method	Accuracy (%)
T5 (3B, fine-tuned)	95.20
Random Sampling (5 runs)	87.95 $\pm$ 2.74
$k\text{NN}_{\text{roberta}}$ (majority voting)	50.20
$\text{KATE}_{\text{roberta}}$	91.99
$\text{KATE}_{\text{nli}}$	90.40
$\text{KATE}_{\text{nli+sts-b}}$	90.20
$\text{KATE}_{\text{sst-2}}$	93.43

Key observations:

$\text{KATE}_{\text{roberta}}$ improves accuracy by 4.04 percentage points over random prompt sampling while eliminating variance caused by random selection.
When the sentence encoder is fine-tuned on the source task dataset ( $\text{KATE}_{\text{sst-2}}$ ), accuracy reaches 93.43%, approaching the 95.20% accuracy of a fully fine-tuned T5-3B model.
The standalone nearest-neighbor classification baseline ( $k\text{NN}_{\text{roberta}}$ ) achieves 50.20% (random guess baseline), demonstrating that retrieval alone is insufficient and acts synergistically with GPT-3's generation capabilities.

Knowl 5 — Table-to-Text Generation Performance on ToTTo

data/table

On the controlled table-to-text benchmark ToTTo (with $k=2$ in-context demonstrations due to context window constraints), KATE was evaluated against random sampling, standalone $k\text{NN}$ , and a supervised fine-tuned T5-3B model on the development set across the overall, overlap, and non-overlap subsets using BLEU and PARENT metrics.

Method	Overall		Overlap Subset		Nonoverlap Subset
	BLEU	PARENT	BLEU	PARENT	BLEU	PARENT
T5 (3B, fine-tuned)	41.2	53.0	46.7	56.1	35.8	50.0
Random Sampling	28.4 $\pm$ 2.1	39.3 $\pm$ 2.6	31.2 $\pm$ 2.5	41.8 $\pm$ 3.0	25.6 $\pm$ 1.8	37.0 $\pm$ 2.3
$k\text{NN}_{\text{roberta}}$	14.1	12.6	20.1	17.9	8.0	7.52
$\text{KATE}_{\text{roberta}}$	41.0	50.6	48.4	55.9	33.6	45.5
$\text{KATE}_{\text{nli}}$	39.9	49.5	47.4	54.6	32.5	44.5
$\text{KATE}_{\text{nli+sts-b}}$	38.8	48.2	46.2	53.1	31.5	43.4

$\text{KATE}_{\text{roberta}}$ yields a +12.6 BLEU and +11.3 PARENT gain over random sampling overall, performing on par with the fine-tuned T5-3B baseline (41.0 vs. 41.2 BLEU). Furthermore, KATE reduces factual hallucination by retrieving tables with matching schemas and numeric formats, providing structural templates for generation.

Knowl 6 — Open-Domain Question Answering Benchmarks Across NQ, WQ, and TriviaQA

data/table

KATE was evaluated on open-domain question answering benchmarks using Exact Match (EM) metric after string normalization against zero/few-shot baselines and supervised fine-tuned models. The prompt setup used $k=64$ retrieved demonstrations for Natural Questions (NQ) and WebQuestions (WQ), and $k=10$ demonstrations for TriviaQA (due to token length limits).

Method	NQ	WQ	TriviaQA
RAG (Open-Domain)	44.5	45.5	68.0
T5+SSM (Closed-Book, 11B)	36.6	44.7	60.5
T5 (Closed-Book, 11B)	34.5	37.4	50.1
GPT-3 (64 random examples)	29.9	41.5	–
Random Sampling	28.6 $\pm$ 0.3	41.0 $\pm$ 0.5	59.2 $\pm$ 0.4
$k\text{NN}_{\text{roberta}}$	24.0	23.9	26.2
$\text{KATE}_{\text{roberta}}$	40.0	47.7	57.5
$\text{KATE}_{\text{nli}}$	40.8	50.6	60.9
$\text{KATE}_{\text{nli+sts-b}}$	41.6	50.2	62.4

$\text{KATE}_{\text{nli+sts-b}}$ outperforms closed-book fine-tuned T5-11B on all three datasets (e.g., 41.6 vs. 34.5 EM on NQ; 50.2 vs. 37.4 EM on WQ) and improves upon standard few-shot GPT-3 with random sampling by +13.0 EM on NQ, +9.2 EM on WQ, and +3.2 EM on TriviaQA.

Knowl 7 — Task Alignment Principle for Sentence Retriever Selection
empirical result
The choice of sentence encoder $\mu_\theta$ in KATE governs prompt selection quality according to the semantic alignment between the encoder's fine-tuning objective and the downstream task:
1. Semantically Aligned Fine-Tuning: On Question Answering tasks (NQ, WQ, TriviaQA), sentence encoders fine-tuned on Natural Language Inference (SNLI, MultiNLI) and Semantic Textual Similarity (STS-B) outperform un-fine-tuned RoBERTa-large (e.g., NQ accuracy increases from 40.0% with $\text{KATE}_{\text{roberta}}$ to 41.6% with $\text{KATE}_{\text{nli+sts-b}}$ ) because STS/NLI objectives train the encoder to identify semantic and syntactic equivalence across questions.
2. Task-Specific Alignment: On sentiment classification, fine-tuning the retriever directly on the SST-2 sentiment task ( $\text{KATE}_{\text{sst-2}}$ ) improves IMDB test accuracy from 91.99% ( $\text{KATE}_{\text{roberta}}$ ) to 93.43%.
3. Mismatched Fine-Tuning Degradation: On table-to-text generation (ToTTo) and sentiment analysis, fine-tuning the retriever on unrelated NLI/STS-B datasets slightly degrades performance compared to un-fine-tuned RoBERTa-large (ToTTo overall BLEU drops from 41.0 with $\text{KATE}_{\text{roberta}}$ to 39.9 with $\text{KATEnli}$ and 38.8 with $\text{KATEnli+sts-b}$ ), demonstrating that fine-tuning on dissimilar objectives impairs retrieval of appropriate task-specific formatting.
Knowl 8 — Scaling Characteristics of In-Context Example Count and Retrieval Candidate Pool Size
empirical result
Evaluating KATE under varying hyperparameters on the Natural Questions (NQ) dataset reveals two scaling dynamics:
- In-Context Example Count ( $k$ ): Testing $k \in \{5, 10, 20, 35, 64\}$ , both random sampling and KATE improve monotonically as $k$ increases. However, KATE consistently maintains a large performance margin at all counts; even at $k=5$ , $\text{KATE}_{\text{nli+sts-b}}$ achieves $\approx 37\%$ EM, outperforming 64 randomly sampled examples ( $\\approx 29\%$ EM).
- Datastore Pool Size ( $N$ ): Testing training datastore pool sizes of $N \in \{1\text{k}, 2\text{k}, 5\text{k}, 10\text{k}, 30\text{k}, 70\text{k}\}$ with fixed $k=64$ , KATE EM scores scale upwards monotonically with pool size (increasing from $\approx 28\%$ at $1\text{k}$ to $41.6\%$ at $70\text{k}$ ). In contrast, random sampling performance remains flat across all candidate datastore sizes ( $\\approx 28.6\%$ EM), as increasing the pool size only benefits nearest-neighbor retrieval by providing higher-similarity demonstrations.

Knowl 9 — Robustness of KATE to In-Context Demonstration Permutations

empirical result

The sensitivity of KATE to the ordering of the $k$ retrieved demonstration examples was evaluated on the Natural Questions dataset using $\text{KATE}_{\text{nli+sts-b}}$ :

Across three random permutations, default order (descending similarity $s(x_{\sigma(i)}, x) \ge s(x_{\sigma(j)}, x)$ for $i < j$ ), and reverse order (ascending similarity), the Exact Match score ranges between 41.6% and 42.8% (a variation of $\le 1.2$ points). This variation is negligible compared to the $\approx 13.0$ point gap between KATE and random example selection, demonstrating that example relevance is the dominant factor rather than example order.

Knowl 10 — Performance Disparity Between Classification and Generation Tasks in Retrieval-Augmented In-Context Learning
limitation
While KATE-augmented in-context learning with GPT-3 matches or surpasses supervised fine-tuned models (such as T5-3B and T5-11B) on long-form text generation (ToTTo table-to-text) and open-domain question answering, it continues to lag slightly behind supervised fine-tuned baselines on short sequence classification tasks like sentiment analysis (93.43% for $\text{KATE}_{\text{sst-2}}$ vs. 95.20% for fine-tuned T5-3B on IMDB). This indicates that in-context example retrieval provides its greatest relative advantages on structured text generation where syntactic and semantic templates are essential.

Coverage note — None was omitted; all contributed methods, empirical findings, ablations, baseline comparisons, and limitations are fully covered.

References

1.Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544.
2.Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326.
3.Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
4.Deng Cai, Yan Wang, Wei Bi, Zhaopeng Tu, Xiaojiang Liu, Wai Lam, and Shuming Shi. 2018. Skeleton-to-response: Dialogue generation guided by retrieval memory. arXiv preprint arXiv:1809.05296.
5.Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2017. Faithful to the original: Fact aware neural abstractive summarization. arXiv preprint arXiv:1711.04434.
6.Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055.
7.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
8.Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William W Cohen. 2019. Handling divergent reference texts when evaluating table-to-text generation. arXiv preprint arXiv:1906.01081.
9.Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723.
10.Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. 2018. Search engine guided neural machine translation. In AAAI, pages 5133–5140.
11.Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. 2018. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437–450.
12.Tatsunori B Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S Liang. 2018. A retrieve-and-edit framework for predicting structured outputs. In Advances in Neural Information Processing Systems, pages 10052–10062.
13.Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
14.Frank Jakel, Bernhard Scholkopf, and Felix A Wichmann. 2008. Generalization and similarity in exemplar models of categorization: Insights from machine learning. Psychonomic Bulletin & Review, 15(2):256–271.
15.Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551.
16.Vladimir Karpukhin, Barlas Oguz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906.
17.Nora Kassner and Hinrich Schutze. 2020. Bertknn: Adding a knn search component to pretrained language models for better qa. arXiv preprint arXiv:2005.00766.
18.Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Nearest neighbor machine translation. arXiv preprint arXiv:2010.00710.
19.Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172.
20.Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466.
21.Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291.
22.Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
23.Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv preprint arXiv:2005.11401.
24.Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: A simple approach to sentiment and style transfer. arXiv preprint arXiv:1804.06437.
25.Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
26.Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142–150.
27.Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2020. Generation-augmented retrieval for open-domain question answering. arXiv preprint arXiv:2009.08553.
28.Gaurav Pandey, Danish Contractor, Vineet Kumar, and Sachindra Joshi. 2018. Exemplar encoder-decoder for neural conversation generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1329–1338.
29.Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
30.Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of EMNLP.
31.Hao Peng, Ankur P Parikh, Manaal Faruqui, Bhuwan Dhingra, and Dipanjan Das. 2019. Text generation with exemplar-based adaptive decoding. arXiv preprint arXiv:1904.04428.
32.Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
33.Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
34.Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
35.Nazneen Fatema Rajani, Ben Krause, Wengpeng Yin, Tong Niu, Richard Socher, and Caiming Xiong. 2020. Explaining and improving model behavior with k nearest neighbor representations. arXiv preprint arXiv:2010.09030.
36.Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
37.Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. arXiv preprint arXiv:2004.09813.
38.Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980.
39.Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642.
40.Yiping Song, Rui Yan, Xiang Li, Dongyan Zhao, and Ming Zhang. 2016. Two are better than one: An ensemble of retrieval-and generation-based dialog systems. arXiv preprint arXiv:1610.07149.
41.Eiichiro Sumita and HDA Hitoshi. 1991. Experiments and prospects of example-based machine translation. In 29th Annual Meeting of the Association for Computational Linguistics, pages 185–192.
42.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30:5998–6008.
43.Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
44.Jason Weston, Emily Dinan, and Alexander H Miller. 2018. Retrieve and refine: Improved sequence generation models for dialogue. arXiv preprint arXiv:1808.04776.
45.Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426.
46.Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
47.Yu Wu, Furu Wei, Shaohan Huang, Yunli Wang, Zhoujun Li, and Ming Zhou. 2019. Response generation by context-aware prototype editing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7281–7288.
48.Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934.
49.Rui Yan, Yiping Song, and Hua Wu. 2016. Learning to respond with deep neural networks for retrieval-based human-computer conversation system. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 55–64.
50.Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5753–5763.
51.Tony Z Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. arXiv preprint arXiv:2102.09690.
52.Morteza Ziyadi, Yuting Sun, Abhishek Goswami, Jade Huang, and Weizhu Chen. 2020. Example-based named entity recognition. arXiv preprint arXiv:2008.10570.

Citation

MLA

Liu, J., et al. “What Makes Good In-Context Examples for GPT-3?”. Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, 2022, pp. 100–14, https://doi.org/10.18653/v1/2022.deelio-1.10.

APA

Liu, J., Shen, D., Zhang, Y., Dolan, W. B., Carin, L., & Chen, W. (2022). What Makes Good In-Context Examples for GPT-3?. Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, 100–114. https://doi.org/10.18653/v1/2022.deelio-1.10

Chicago

Liu, J., D. Shen, Y. Zhang, W. B. Dolan, L. Carin, and W. Chen. 2022. “What Makes Good In-Context Examples for GPT-3?”. Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, 100–114. https://doi.org/10.18653/v1/2022.deelio-1.10.

Harvard

Liu, J. et al. (2022) “What Makes Good In-Context Examples for GPT-3?”, Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures. Association for Computational Linguistics, pp. 100–114. Available at: https://doi.org/10.18653/v1/2022.deelio-1.10.

Vancouver

1. Liu J, Shen D, Zhang Y, Dolan WB, Carin L, Chen W (2022) What Makes Good In-Context Examples for GPT-3?. In: Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures. Association for Computational Linguistics, pp 100–114

BibTeX

@inproceedings{liu-etal-2022-makes,
    title = "What Makes Good In-Context Examples for {GPT}-3?",
    author = "Liu, Jiachang  and
      Shen, Dinghan  and
      Zhang, Yizhe  and
      Dolan, Bill  and
      Carin, Lawrence  and
      Chen, Weizhu",
    editor = "Agirre, Eneko  and
      Apidianaki, Marianna  and
      Vuli{\'c}, Ivan",
    booktitle = "Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures",
    month = may,
    year = "2022",
    address = "Dublin, Ireland and Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.deelio-1.10/",
    doi = "10.18653/v1/2022.deelio-1.10",
    pages = "100--114"
}

Metadata:ACL Anthology

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Order Setting	Trial 1	Trial 2	Trial 3	Default (Most Similar First)	Reverse (Most Similar Last)
EM Score (%)	42.0	42.5	42.0	41.6	42.8

What Makes Good In-Context Examples for GPT-3?

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — KATE: kNN-Augmented In-Context Example Selection

Knowl 2 — Conditional Autoregressive Formulation of In-Context Generation

Knowl 3 — Correlation Between Prompt Semantic Distance and In-Context Generation Accuracy

Knowl 4 — Cross-Dataset Sentiment Analysis Evaluation Under KATE

Knowl 5 — Table-to-Text Generation Performance on ToTTo

Knowl 6 — Open-Domain Question Answering Benchmarks Across NQ, WQ, and TriviaQA

Knowl 7 — Task Alignment Principle for Sentence Retriever Selection

Knowl 8 — Scaling Characteristics of In-Context Example Count and Retrieval Candidate Pool Size

Knowl 9 — Robustness of KATE to In-Context Demonstration Permutations

Knowl 10 — Performance Disparity Between Classification and Generation Tasks in Retrieval-Augmented In-Context Learning

References

Citation

BibTeX

Access the Paper