Back

RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs

Afra Feyza AkyürekEkin AkyürekAshwin KalyanPeter ClarkDerry Tanti WijayaNiket Tandon

article2023ACL139 citations

Presents RL4F, a multi-agent framework that trains a compact critique generator with reinforcement learning to produce natural language feedback that guides frozen, black-box language models like GPT-3 to correct their own output errors without requiring fine-tuning.

Listen

Large language models often make factual, logical, or structural errors when generating text. While models can refine their predictions when provided with natural language feedback, obtaining human-written critiques in real time is too costly and slow for operational use. Furthermore, many enterprise applications rely on closed, third-party black-box models that cannot be directly modified or fine-tuned. To address these constraints, the article introduces RL4F (Reinforcement Learning for Feedback Generation), a collaborative system where a smaller, dedicated critique model is trained to provide plain-language feedback that guides a fixed, frozen downstream task model to correct its own errors.

The framework pairs a relatively lightweight critique model (a 770-million-parameter T5 network) with a large, fixed language model (175-billion-parameter GPT-3). The critique model is first warm-started on example critiques and then optimized using Proximal Policy Optimization, a reinforcement learning method. Training directly rewards the critic when its generated feedback steers the large language model to produce a more accurate final answer. This setup was evaluated across three distinct domains: topic-based summarization, everyday action planning (the Interscript benchmark), and a synthetic word alphabetization task.

The evaluation demonstrates that the reinforced critique approach consistently outperforms standard supervised training, retrieval baselines, and self-refinement prompting. In action planning and summarization, RL4F achieved relative text quality and similarity improvements of up to 10% over alternative automated feedback methods, narrowing the gap toward upper-bound performance achieved with human feedback. In alphabetization, the framework raised exact match accuracy to 66.1%, outperforming supervised feedback (38.9%) by more than 27 absolute percentage points and full model fine-tuning (55.3%). Moreover, the critique model scaled effectively with parameter size and produced additional accuracy gains when applied iteratively across multiple rounds of feedback without corrupting previously correct answers.

These results provide a practical, cost-effective blueprint for improving the reliability of deployed AI systems. Instead of undertaking expensive full-model fine-tuning or paying for continuous human monitoring, organizations can deploy a compact, specialized critic model to act as a lightweight external adapter. This multi-agent strategy reduces operational computing expenses, preserves the general capabilities of the primary system, and maintains interpretable, human-readable communication between models.

Organizations seeking to improve output quality in automated workflows should consider adopting external, reward-tuned feedback models for high-stakes tasks such as multi-step planning and content summarization. Before full production deployment, teams should conduct domain-specific pilot testing to monitor for potential semantic drift—where the critic might invent non-standard phrasing—and establish fallback rules for ambiguous model outputs. Future development should focus on extending the architecture into an ensemble framework capable of integrating feedback from multiple automated models and human experts simultaneously.

Allen Institute for AIBoston UniversityMassachusetts Institute of TechnologyMonash Universitygeneral purposelanguage modelmulti-agentnatural language feedbackreinforcement learningmodel output repair
arXiv: 2305.08844feyzaakyurek/rl4f
  • Paper: Fine-Tuning Language Models from Human Preferences, Daniel M. Ziegler et al. (2019). This foundational preference-optimization work introduces reward-model-guided reinforcement learning for language models, the training logic RL4F adapts to optimize a critic for a fixed downstream model.
  • Paper: Learning to summarize from human feedback, Nisan Stiennon et al. (2020). Its human-feedback summarization pipeline establishes reward modeling and reinforcement learning for language generation, directly contextualizing RL4F’s use of downstream task performance to train feedback generation.
  • Paper: Guiding Large Language Models via Directional Stimulus Prompting, Zekun Li et al. (2023). Directional Stimulus Prompting trains a small policy with reinforcement learning against a frozen black-box model’s task performance, a close precursor to RL4F’s feedback generator for an unmodifiable target.
Cover for RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs

Abstract

Despite their unprecedented success, even the largest language models make mistakes. Similar to how humans learn and improve using feedback, previous work proposed providing language models with natural language feedback to guide them in repairing their outputs. Because human-generated critiques are expensive to obtain, researchers have devised learned critique generators in lieu of human critics while assuming one can train downstream models to utilize generated feedback. However, this approach does not apply to black-box or limited access models such as ChatGPT, as they cannot be fine-tuned. Moreover, in the era of large general-purpose language agents, fine-tuning is neither computationally nor spatially efficient as it results in multiple copies of the network. In this work, we introduce RL4F (Reinforcement Learning for Feedback), a multi-agent collaborative framework where the critique generator is trained to maximize end-task performance of GPT-3, a fixed model more than 200 times its size. RL4F produces critiques that help GPT-3 revise its outputs. We study three datasets for action planning, summarization and alphabetization and show relative improvements up to 10% in multiple text similarity metrics over other learned, retrieval-augmented or prompting-based critique generators.

Topics

general purposelanguage modelmulti-agentreinforcement learninglarge language models

Keywords

natural language feedbackreinforcement learningmodel output repairpolicy gradient learningdiscrete prompt learningsemantic driftcritique generators

Table of Contents

  • 1 Introduction
  • 2 Related Works
  • 2.1 What kind of feedback is used and where does it originate?
  • 2.2 How is feedback used?
  • 2.3 Adapters & Discrete Prompt Learning
  • 3 Background
  • 3.1 SUPERVISED: Supervised Learning for Critique Generation
  • 3.2 Direct-Refinement
  • 4 RL4F: Reinforcement Learning for Feedback Generation
  • 5 Datasets
  • 5.1 Topic-Based Summarization
  • 5.2 Interscript
  • 5.3 Synthetic Task: Alphabetization
  • 6 Experiments and Results
  • 6.1 Planning and Summarization
  • 6.2 Alphabetization
  • 7 Analysis
  • 8 Conclusion
  • 9 Limitations
  • Acknowledgement
  • References
  • A Dataset Processing
  • B Experiment Details
  • B.1 Data Formats
  • B.2 Prompts for GPT-3
  • B.3 Standard Deviations
  • B.4 Hyperparameters
  • B.5 Results for Self-Refine (Madaan et al., 2023) on Alphabetization
  • C Sample Predictions
  • D Learning Curves for Reinforcement Learning
  • ACL 2023 Responsible NLP Checklist

Knowls

  1. Knowl 1 — RL4F trains a critique generator to improve a frozen task model

    model/method

    RL4F separates critique generation from task execution. A fixed task model LMtaskLM_{\text{task}} maps an input xx to an initial prediction y^\hat{y}; a separate, trainable critique model LMcritiqueLM_{\text{critique}} takes (x,y^)(x,\hat{y}) and generates natural-language feedback c^\hat{c}. The same task model then produces a revised prediction y^new\hat{y}_{\text{new}} conditioned on the input, initial prediction, and critique. Training changes only LMcritiqueLM_{\text{critique}}: the task model’s parameters remain frozen, so it can be treated as a black-box model accessed through prompts. RL4F’s distinctive objective is to train the critique generator for the downstream task model’s final performance, rather than only for resemblance to human-written critiques.

  2. Knowl 2 — RL4F warm-starts critique generation with supervision, then optimizes it with PPO

    algorithm

    RL4F takes task examples containing inputs xx, ground-truth outputs yy, and initial task-model predictions y^\hat{y}; an initialized critique generator; a frozen task model; and in-context examples for refinement. It returns a critique generator adapted to improve the task model’s outputs.

    Input: Task data with xx, ground truth yy, and initial prediction y^\hat{y}
    Input: Critique model LMcritiqueLM_{\text{critique}} and frozen task model LMtaskLM_{\text{task}}
    Input: Task-model in-context examples for refinement
    Warm-start LMcritiqueLM_{\text{critique}} by supervised learning on critiques conditioned on (x,y^)(x, \hat{y})
    Repeat until convergence:
        Sample a mini-batch of task examples
        Generate critiques c^\hat{c} from LMcritique(x,y^)LM_{\text{critique}}(x, \hat{y}) in parallel
        Ask the frozen LMtaskLM_{\text{task}} to revise each prediction using (x,y^,c^)(x, \hat{y}, \hat{c})
        Score each revised output against its ground truth using the task-specific reward
        Apply PPO with KL regularization to update LMcritiqueLM_{\text{critique}}
    Return LMcritiqueLM_{\text{critique}}

    The supervised warm start maximizes the likelihood of human or synthetic critiques given (x,y^)(x,\hat{y}). PPO then updates the critique generator using the reward from the revised task-model output; reward is assigned only at the end of generation, when the model emits an end-of-sentence token or reaches its token limit. The paper’s common PPO settings include entropy coefficient 0.0010.001, discount factor 0.990.99, GAE parameter 0.950.95, clipping ratio 0.20.2, and value-function coefficient 0.50.5. The main per-task PPO settings were: alphabetization, batch size 2424, five epochs per update, learning rate 10−610^{-6}, initial KL coefficient 10−510^{-5}, target KL 33; Interscript, batch size 88, five epochs per update, learning rate 5×10−75\times10^{-7}, initial KL coefficient 0.010.01, target KL 22; summarization, batch size 44, three epochs per update, learning rate 10−710^{-7}, initial KL coefficient 0.010.01, target KL 22. The task-model weights are never updated.

  3. Knowl 3 — RL4F uses task-specific terminal rewards for planning, summarization, and sorting

    equation

    For action planning and topic-based summarization, RL4F rewards the revised task-model output using the mean of ROUGE-1, ROUGE-2, and ROUGE-L against the ground truth. For alphabetization, the reward is one minus normalized word-level Levenshtein distance:

    R(y^,y)=1−Levenshtein⁡(y^,y)max⁡(∣y^∣,∣y∣)R(\hat{y},y)=1-\frac{\operatorname{Levenshtein}(\hat{y},y)}{\max(|\hat{y}|,|y|)}

    Here y^\hat{y} is a revised predicted output, yy is its ground-truth output, and ∣⋅∣|\cdot| is the number of words in a list; the edit distance counts word-level rather than character-level operations. An exactly sorted list receives reward 11. For the text-generation tasks, the reward is mean⁡(R1(y^,y),R2(y^,y),RL(y^,y))\operatorname{mean}(R_1(\hat{y},y),R_2(\hat{y},y),R_L(\hat{y},y)), where R1R_1, R2R_2, and RLR_L denote the ROUGE-1, ROUGE-2, and ROUGE-L scores. These rewards score the outcome after refinement, not the critique’s similarity to a reference critique.

  4. Knowl 4 — Evaluation uses a prompted GPT-3 task model and three datasets with different data regimes

    experimental setup

    The task model was GPT-3 code-davinci-002, used through prompting and left frozen; the paper reports that only the largest, 175-billion-parameter GPT-3 model could perform the refinement task. The critique generator was initialized from T5-large. GPT-3 prompts contained 3 hand-written in-context examples for Interscript, 1 for summarization, and 6 for alphabetization, constrained by a 4096-token input limit. GPT-3 prompting used temperature 00, except that initial alphabetization predictions were sampled at temperature 0.50.5.

    The topic-based summarization dataset contains 14,230 training, 1,150 validation, and 2,658 test tuples, with passages, questions, summaries, initial predictions, and human critiques. The Interscript action-planning data used in the experiments contains 253 training, 45 validation, and 169 test examples, each with 1–4 reference texts; the authors used a subset with distractors removed. For alphabetization, the authors generated lists of 3–12 words from a 43,000-word English lexicon. They made critique-training examples by applying reorder, replace, add, repeat, or remove distortions to sorted lists, while leaving some lists unchanged with a critique stating they were correctly sorted. They used 40,000 examples for supervised warm-starting and 10,000, 1,000, and 1,000 examples for the PPO train, development, and test splits, respectively.

  5. Knowl 5 — RL4F improves planning and summarization results over learned and retrieval-based critique baselines

    data/table

    The table reports final task-model outputs under different critique sources. BLEURT and BERTScore are reported in the paper’s scales; ROUGE values are out of 100 and are shown as ROUGE-1/ROUGE-2/ROUGE-L. RL4F obtains the strongest score among the non-human critique methods in every summarization column and in Interscript BLEURT and ROUGE-1/ROUGE-L; MemPrompt has the highest Interscript BERTScore, and Direct-Refinement has the highest Interscript ROUGE-2. Human-written critiques perform better than all automatic methods.

    Interscript results (BLEURT; BERTScore; ROUGE-1/2/L): Direct-Refinement, −1.07-1.07; 86.9786.97; 15.8/0.9/15.515.8/0.9/15.5. SUPERVISED, −1.02-1.02; 86.9986.99; 19.4/0.5/18.519.4/0.5/18.5. MemPrompt, −1.18-1.18; 87.4587.45; 16.9/1.9/16.716.9/1.9/16.7. RL4F, −0.92-0.92; 87.2387.23; 22.1/0.9/21.322.1/0.9/21.3. Gold feedback, −0.69-0.69; 89.5689.56; 40.7/6.8/39.140.7/6.8/39.1.

    Topic-based summarization results (BLEURT; BERTScore; ROUGE-1/2/L): Direct-Refinement, 0.090.09; 93.193.1; 54.3/46.0/50.954.3/46.0/50.9. SUPERVISED, 0.060.06; 92.992.9; 53.2/46.4/50.753.2/46.4/50.7. MemPrompt, 0.090.09; 91.991.9; 48.8/40.4/45.648.8/40.4/45.6. RL4F, 0.100.10; 93.693.6; 55.1/48.2/52.655.1/48.2/52.6. Gold feedback, 0.220.22; 94.294.2; 58.3/50.3/55.858.3/50.3/55.8. The results show that feedback trained against downstream performance can outperform the supervised critique generator and retrieval baseline, while still falling short of human feedback.

  6. Knowl 6 — On alphabetization, RL4F beats supervised critiques and slightly exceeds direct refinement

    data/table

    The alphabetization evaluation reports exact-match accuracy as a percentage and inverse Levenshtein score on a 00–11 scale. The initial GPT-3 predictions score 63.763.7 exact match and 0.910.91 inverse Levenshtein. Fine-tuning davinci scores 55.355.3 and 0.890.89; MemPrompt scores 57.857.8 and 0.890.89; Direct-Refinement scores 65.965.9 and 0.920.92; SUPERVISED critiques score 38.938.9 and 0.820.82; RL4F scores 66.166.1 and 0.920.92; and gold feedback scores 75.975.9 and 0.940.94.

    Thus RL4F improves exact match by 27.227.2 percentage points over SUPERVISED, and by 2.42.4 points over the initial predictions. Its exact-match advantage over Direct-Refinement is only 0.20.2 points, while both methods have an inverse Levenshtein score of 0.920.92. Gold feedback remains substantially better. The authors attribute the supervised model’s poor performance in part to synthetic critiques and errors that do not necessarily match the task model’s own mistakes.

  7. Knowl 7 — Larger critique generators improve RL4F performance on Interscript

    empirical result

    The scaling experiment compares T5-small, T5-base, and T5-large critique generators on Interscript, spanning approximately 60 million to 770 million parameters. Performance is the mean of ROUGE-1, ROUGE-2, and ROUGE-L for generated plans. RL4F’s score rises substantially as critique-generator size increases, whereas the SUPERVISED baseline does not show the same increasing trend: it rises from T5-small to T5-base and then falls at T5-large. This is evidence of a positive scaling trend for RL4F in this experiment, not a claim established across other tasks or model families.

  8. Knowl 8 — Iterative RL4F feedback corrects additional alphabetization outputs

    empirical result

    In an alphabetization experiment, the authors repeatedly sampled RL4F critiques and applied the resulting refinements to the current predictions. The plotted run reaches seven corrected samples at some iterations, which the authors describe as up to seven more corrections through iterative use. Correct examples were not removed from later rounds, so a critique could also damage an already-correct ordering; RL4F’s discretion to indicate that a list is already sorted matters in this setting. By contrast, the authors observed that iterative Direct-Refinement sometimes scrambled an already-correct ordering without gaining new correct examples.

  9. Knowl 9 — Critique fluency varies by task, and summarization feedback can be generic

    empirical result

    The authors’ manual inspection found that close to 100% of RL4F critiques for alphabetization and action planning were grammatical. They also report that most sampled critiques were intelligible and observed minimal signs of semantic drift in fluency and naturalness. Summarization critiques were less consistently useful: they sometimes repeated themselves or offered generic demands to fix an answer. The authors speculate that the task model’s preference for natural-language inputs may help preserve naturalness, but state that this explanation requires further investigation.

  10. Knowl 10 — RL4F does not explicitly prevent semantic drift or support multiple feedback experts

    limitation

    RL4F optimizes downstream task performance without an explicit constraint that generated critiques remain natural or interpretable. The authors caution that semantic drift could therefore undermine interpretability even when task performance improves. The framework also provides no explicit mechanism for incorporating newly available critique labels or combining feedback from multiple experts, such as humans and other models. Finally, the experiments cover GPT-3 in a setting where task-model training is inefficient or impossible; the authors note that treating the task model as fixed may be a conservative assumption in other settings.

Coverage note — No substantial contributed material was omitted; appendix examples and learning curves are supporting illustrations of the included methods and evaluations.

References

  1. 1.Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
  2. 2.Andreas Blank. 1999. Why do new meanings occur? a cognitive typology of the motivations for lexical semantic change andreas blank. Cognitive Linguistics Research, page 61.
  3. 3.Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  4. 4.Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128.
  5. 5.Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P Xing, and Zhiting Hu. 2022. Rlprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint arXiv:2205.12548.
  6. 6.David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A Saurous, Jascha Sohl-Dickstein, et al. 2022. Language model cascades. arXiv preprint arXiv:2207.10342.
  7. 7.Felix Faltings, Michel Galley, Gerold Hintz, Chris Brockett, Chris Quirk, Jianfeng Gao, and Bill Dolan. 2021. Text editing by command. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5259–5274, Online. Association for Computational Linguistics.
  8. 8.Patrick Fernandes, Aman Madaan, Emmy Liu, António Farinhas, Pedro Henrique Martins, Amanda Bertsch, José GC de Souza, Shuyan Zhou, Tongshuang Wu, Graham Neubig, et al. 2023. Bridging the gap: A survey on integrating (human) feedback for natural language generation. arXiv preprint arXiv:2305.00955.
  9. 9.Ge Gao, Eunsol Choi, and Yoav Artzi. 2022. Simulating bandit learning from user feedback for extractive question answering. arXiv preprint arXiv:2203.10079.
  10. 10.Stephen P Harter. 1975. A probabilistic approach to automatic keyword indexing. part i. on the distribution of specialty words in a technical literature. Journal of the american society for information science, 26(4):197–206.
  11. 11.Keith Vertanen. 2018. Big english word lists.
  12. 12.Jason Lee, Kyunghyun Cho, and Douwe Kiela. 2019. Countering language drift via visual grounding. arXiv preprint arXiv:1909.04499.
  13. 13.Jared Lichtarge, Chris Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar, and Simon Tong. 2019. Corpora generation for grammatical error correction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3291–3301, Minneapolis, Minnesota. Association for Computational Linguistics.
  14. 14.Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  15. 15.Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He, Sean Welleck, Hannaneh Hajishirzi, and Yejin Choi. 2022. Rainier: Reinforced knowledge introspector for commonsense question answering. arXiv preprint arXiv:2210.03078.
  16. 16.Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. 2022. Memory-assisted prompt editing to improve gpt-3 after deployment. arXiv preprint arXiv:2201.06009.
  17. 17.Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651.
  18. 18.Jonathan Mallinson, Aliaksei Severyn, Eric Malmi, and Guillermo Garrido. 2020. FELIX: Flexible text editing through tagging and insertion. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1244–1255, Online. Association for Computational Linguistics.
  19. 19.Eric Malmi, Yue Dong, Jonathan Mallinson, Aleksandr Chuklin, Jakub Adamek, Daniil Mirylenka, Felix Stahlberg, Sebastian Krause, Shankar Kumar, and Aliaksei Severyn. 2022. Text generation with text-editing models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorial Abstracts, pages 1–7, Seattle, United States. Association for Computational Linguistics.
  20. 20.Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2022a. Fast model editing at scale. In International Conference on Learning Representations.
  21. 21.Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. 2022b. Memory-based model editing at scale. In International Conference on Machine Learning, pages 15817–15831. PMLR.
  22. 22.Shikhar Murty, Christopher D Manning, Scott Lundberg, and Marco Tulio Ribeiro. 2022. Fixing model bugs with natural language patches. arXiv preprint arXiv:2211.03318.
  23. 23.Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155.
  24. 24.Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. Adapterhub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020): Systems Demonstrations, pages 46–54, Online. Association for Computational Linguistics.
  25. 25.Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67.
  26. 26.Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2022. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint arXiv:2210.01241.
  27. 27.Machel Reid and Graham Neubig. 2022. Learning to model editing processes. arXiv preprint arXiv:2205.12374.
  28. 28.Joana Ribeiro, Shashi Narayan, Shay B. Cohen, and Xavier Carreras. 2018. Local string transduction as sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1360–1371, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  29. 29.William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802.
  30. 30.Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2022. Training language models with natural language feedback. arXiv preprint arXiv:2204.14146.
  31. 31.Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. 2022. PEER: A collaborative language model. arXiv preprint arXiv:2208.11663.
  32. 32.Timo Schick, Sahana Udupa, and Hinrich Schütze. 2021. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP. Transactions of the Association for Computational Linguistics, 9:1408–1424.
  33. 33.John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
  34. 34.Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics.
  35. 35.Weiyan Shi, Emily Dinan, Kurt Shuster, Jason Weston, and Jing Xu. 2022. When life gives you lemons, make cherryade: Converting feedback from bad responses into good labels. arXiv preprint arXiv:2210.15893.
  36. 36.Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1630–1640, Florence, Italy. Association for Computational Linguistics.
  37. 37.Niket Tandon, Aman Madaan, Peter Clark, Keisuke Sakaguchi, and Yiming Yang. 2021. Interscript: A dataset for interactive learning of scripts through error feedback. arXiv preprint arXiv:2112.07867.
  38. 38.Niket Tandon, Aman Madaan, Peter Clark, and Yiming Yang. 2022. Learning to repair: Repairing model output errors after deployment using a dynamic memory of feedback. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 339–352, Seattle, United States. Association for Computational Linguistics.
  39. 39.Norbert Wiener. 1960. Some moral and technical consequences of automation: As machines learn they may develop unforeseen strategies at rates that baffle their programmers. Science, 131(3410):1355–1358.
  40. 40.Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229–256.
  41. 41.Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.

Citation

MLA
Akyürek, A. F., et al. “RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs”. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 7716–33, https://doi.org/10.18653/v1/2023.acl-long.427.
APA
Akyürek, A. F., Akyürek, E., Kalyan, A., Clark, P., Wijaya, D. T., & Tandon, N. (2023). RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 7716–7733. https://doi.org/10.18653/v1/2023.acl-long.427
Chicago
Akyürek, A. F., E. Akyürek, A. Kalyan, P. Clark, D. T. Wijaya, and N. Tandon. 2023. “RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs”. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 7716–33. https://doi.org/10.18653/v1/2023.acl-long.427.
Harvard
Akyürek, A.F. et al. (2023) “RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs”, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pp. 7716–7733. Available at: https://doi.org/10.18653/v1/2023.acl-long.427.
Vancouver
1. Akyürek AF, Akyürek E, Kalyan A, Clark P, Wijaya DT, Tandon N (2023) RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pp 7716–7733

BibTeX

@inproceedings{akyurek-etal-2023-rl4f,
    title = "{RL}4{F}: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs",
    author = "Akyurek, Afra Feyza  and
      Akyurek, Ekin  and
      Kalyan, Ashwin  and
      Clark, Peter  and
      Wijaya, Derry Tanti  and
      Tandon, Niket",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.427/",
    doi = "10.18653/v1/2023.acl-long.427",
    pages = "7716--7733"
}
Metadata:ACL Anthology

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: https://creativecommons.org/licenses/by/4.0/