Learning Diverse Attacks on Large Language Models for Robust Red-Teaming and Safety Tuning

Seanie Lee${}^1$ ${}^{\ast }$ Minsu Kim${}^{1,2,3}$ Lynn Cherif${}^{2,4}$ David Dobre${}^{2,3}$ Juho Lee${}^1$ Sung Ju Hwang${}^1$ Kenji Kawaguchi${}^5$ Gauthier Gidel${}^{2,3,7}$ Yoshua Bengio${}^{2,3,7}$ Esmeralda S. Whitammer${}^6$ Moksh Jain${}^{2,3}$
${}^1$ KAIST ${}^2$ Mila – Québec AI Institute ${}^3$ Université de Montréal ${}^4$ McGill University ${}^5$ National University of Singapore ${}^6$ University of Edinburgh ${}^7$ CIFAR AI Chair

The deployment of large language models (LLMs) in the wild has raised concerns about their potential harmful impacts for nearly a decade ([1, 2]). These concerns have grown with the increasing capabilities of LLMs: even models fine-tuned to satisfy certain safety constraints can be manipulated to produce toxic outputs ([3]). Red-teaming, or identification of 'attack' prompts that elicit undesirable responses, gives model developers as well as regulators a chance to identify and address such vulnerabilities before deployment ([4]). This paper studies the problem of automatically generating diverse attack prompts for LLMs and argues for the potential of robust automated red-teaming in the development of effective defenses.

Effective red-teaming requires identifying many modes of attack ([5]). Methods for automated red-teaming based on stochastic optimization of attack prompts ([6, 7]) have been proposed, while others have used reinforcement learning (RL) to train an attacker language model (LM), allowing to generate novel prompts efficiently at test time ([4, 5]). However, even when regularized to favor diversity, these methods struggle to balance between diversity and effective attacks (Figure 2). They often suffer from mode collapse, where the attacker LM generates a small set of similar prompts, or focus solely on diversity and fail to generate effective attacks (Figure 3). Moreover, we have empirically found that they also fail to discover attacks that transfer across different target LLMs (Table 3).

This paper takes an amortized inference perspective on red-teaming: we view the problem of generating an attack prompt as sampling a latent variable in a probabilistic model. Using the off-policy RL approach of GFlowNet fine-tuning, proposed for inference of linguistic latent variables in ([8]), we fine-tune attack LMs to sample the full posterior distribution over attack prompts.

However, controlling the 'peakiness' of the posterior distribution – the preference of attack quality to attack diversity – is challenging, especially when red-teaming a target LLM that has been safety-tuned to resist some modes of attack, leading to a sparser landscape of attack prompts. Inspired by the success of behavior cloning in offline RL ([9, 10]), we propose a two-stage GFlowNet fine-tuning procedure with MLE smoothing. As illustrated in Figure 1, we first fine-tune a pretrained attacker LM with a GFlowNet objective and collect high-reward attack prompts discovered in the course of training (Step 1). The collected prompts form an offline dataset. Subsequently, the pretrained attacker model is fine-tuned again to maximize the likelihood of the offline dataset (Step 2). The first stage, GFlowNet fine-tuning, enables us to collect a set of diverse and effective attack prompts using exploratory off-policy training. In the second phase, we obtain a smooth distribution over high-reward attack prompts, since all the collected attack prompts in the offline dataset are considered equally important and the attacker LM is trained to maximize their log-likelihood uniformly. Consequently, we find that the attacker LM is able to sample attack prompts that are both diverse and effective.

We empirically evaluate the efficacy of our proposed method in red-teaming five target LLMs: GPT-2 ([11]), Dolly-v2-7b ([12]), Gemma-2b-it ([13]), Llama-2-7b-chat ([14]), and Llama-3.1-8B-Instruct ([15]). Our approach is found to sample more diverse and effective attack prompts than other relevant baselines. Moreover, many of our attack prompts effectively transfer to other target LLMs that are not used for training the attacker model, such as Llama-2-13b/70b-chat, Llama-3-8b/70b-instruct ([15]), Starling-7b-beta ([16]), and Mistral-7b-instruct-v0.2 ([17]). Lastly, we fine-tune a target LLM to generate refusal responses to the discovered attack prompts and find that the model fine-tuned with our red-teaming prompts is more robust than the models safety-tuned with other RL-based red-teaming methods.

It is important to note that while we study an approximate measure of toxicity as a proxy for harmfulness, following past works ([4, 5]), the true harmful impact of an LLM output is often subjective and dependent on the social context of deployment ([2]). We nonetheless believe that the methods we propose will be useful in practice and can be extended to other measures of harmfulness.

As LLMs increase in general capabilities and performance, so does the risk associated to potential misuse of LLMs. To mitigate this, LLMs are often trained to refuse to generate content given prompts that are dangerous, offensive, or harmful ([18, 19]). This is done at various stages of the training process such as filtering out harmful training data ([13]) or fine-tuning on 'safe' responses to harmful prompts ([14]). This process is often augmented by red-teaming, which proactively looks for ways to elicit harmful behavior from models. Prior works ([20, 21, 22]) rely on a large amount of human annotation to identify vulnerabilities of LMs. To automate red-teaming, [4] formulate red teaming as an RL problem and train an LM to sample toxic prompts. However, most RL algorithms are not suitable for sampling diverse objects since they tend to converge to a single reward-maximizing trajectory. To overcome this limitation, [5] propose using a novelty-based reward to encourage a policy to explore diverse samples during RL training. Instead of generating a prompt from scratch, [23] replace words of prompts from a predefined user input pool to attack LMs using Bayesian optimization in a sample-efficient manner. Rainbow Teaming ([24]) samples an attack prompt from a pool and iteratively mutates the prompt with auxiliary LLMs.

Jailbreaks. Jailbreaking and red-teaming are closely related in that red-teaming proactively tries to discover vulnerabilities for the purpose of improving model safety, whereas jailbreaking generally refers to circumventing the built-in safeguards of models. Initially, jailbreaks were found manually through trial and error, taking advantage of the different objectives models were trained against ([3]). Recently, automated jailbreak attacks are becoming increasingly popular. They utilize techniques such as genetic algorithms ([25]), iterative gradient-based methods ([6]), or automated prompting via auxiliary LLMs ([26]) to optimize query prompts. [27] propose a method defending against GCG ([6]), one of the most popular gradient-based jailbreak methods. A drawback of these methods is the computational cost since the optimization has to be performed separately for each new query prompt. Another drawback is the poor transferability of jailbreaks. [28] have shown that prompts optimized by GCG to jailbreak one target LLM do not transfer to jailbreak other target LLMs.

GFlowNets. Generative flow networks (GFlowNets; [29]) are a probabilistic framework to train stochastic policies to sample discrete compositional objects (e.g., graphs, sequences) proportionally to a reward. Sampling objects proportionally to a reward results in diverse high-reward samples. Consequently, GFlowNets have found applications in a wide variety of problems including biological sequence generation ([30]), combinatorial optimization ([31, 32]), Bayesian structure learning ([33]), variational EM with discrete latent variables ([34]), and probabilistic neurosymbolic inference ([35]). Most closely related to our work is ([8]), which uses the GFlowNet objective to fine-tune LMs for solving intractable inference problems such as sampling chains of thought ([36]). We use GFlowNet fine-tuning as a part of our approach for learning policies which generate diverse prompts that elicit toxic responses from target LLMs.

The target LLM, denoted $p_{\varphi }$, samples a text response $\mathbf{y}$ for a given prompt $\mathbf{x}$ with probability $p_{\varphi }(\mathbf{y}\mid \mathbf{x})$. The goal of red-teaming an LLM is to identify prompts $\mathbf{x}$ that elicit toxic responses from the target LLM. A binary toxicity classifier, denoted as $p_{\psi }$, is used to quantify the effectiveness of an attack prompt. Specifically, the effectiveness of a prompt $\mathbf{x}$ is measured by the likelihood of the response $\mathbf{y}\sim p_{\varphi }(\mathbf{y}\mid \mathbf{x})$ being classified as toxic by the classifier: $p_{\psi }(c=1\mid \mathbf{x},\mathbf{y})$, where $c\in \{0,1\}$ is a binary variable denoting toxicity. Moreover, for the attack to be effective, the prompt $\mathbf{x}$ should appear natural, as unnatural prompts (with high perplexity under some prior) are easy to defend against with simple filters ([37]).

Red-teaming can often be a time-consuming process if done manually as the space of prompts is quite large. [4, 5] formulate red-teaming as an RL problem, to automate the discovery of these prompts. This involves training a LM as a policy $p_{\theta }$, parameterized by $\theta$, to generate prompts that maximize the expected reward (as measured by the toxicity of the response generated by the target LLM):

where the KL divergence term, weighted by a hyperparameter $\lambda >0$, encourages the policy $p_{\theta }$ to remain close to an initial pretrained LM $p_{\text{ref}}$, penalizing the generation of prompts $\mathbf{x}$ that are far from natural language text. However, most RL algorithms are not suitable for discovering diverse prompts since they generally concentrate most of probability mass of the policy $p_{\theta }$ on actions with highest reward, often resulting in a deterministic policy that generates a single prompt ([29]). While [5] propose adding a novelty-based reward term along with entropy bonus ([38]) as a regularization to encourage the policy to generate diverse prompts, empirically we find that it is challenging to find an optimal trade-off between diversity and toxicity rate even with the regularization. In the context of red-teaming, identifying diverse and effective attack prompts is critical to ensure that the target LLM is sufficiently safety-tuned for a broad range of scenarios which might be encountered when the model is deployed in the wild.

A probabilistic view of the problem provides a principled alternative. Specifically, problem of generating diverse and effective red-teaming prompts can be viewed as one of generating samples from a (tempered) reward distribution. We adopt the perspective of generative flow networks (GFlowNets; [29, 39]), leveraging their ability to learn policies that sample from a target distribution defined over compositional objects such as sequences ([30]) and graphs ([39]). To instantiate the probabilistic perspective, we propose a two-stage approach designed to learn a stochastic policy to sample diverse and effective prompts for red-teaming. The first stage consists of fine-tuning a pretrained LM $p_{\theta }$ as a GFlowNet policy ([8]) in order to collect prompts, and the second stage restarts fine-tuning from the original pretrained LM policy but this time with maximum likelihood estimation (MLE) on the high-reward prompts collected during GFlowNet training in the first stage.

where $\beta$ and $\gamma$ are positive constants that control the 'peakiness' (tempering) of the toxicity score $R_1(\mathbf{x})$ and of the reference LM likelihood $R_2(\mathbf{x})$, respectively. The prompt $\mathbf{x}=(x_0,x_1,\dots ,x_T)$, consisting of $T$ tokens with a special token $x_0$ indicating the beginning of a sentence, is generated autoregressively from a behavior policy, which is a mix of $p_{\theta }$ and a tempered variant of it. We define $(x_0,x_1,\dots ,x_t)$ as a state in the generative process and the token sampled from the policy at each step is the action. To learn the parameters $\theta$, we use the trajectory balance learning objective ([41]):

As we illustrate in Section 4, using GFlowNet fine-tuning alone to sample effective and diverse red-teaming prompts can be challenging in practice due to non-trivial choice of the temperature parameters $\beta$ and $\gamma$. While in principle there are choices of $\beta$ and $\gamma$ which can balance the reward and diversity well, in practice GFlowNet fine-tuning can be overly sensitive to the peakiness of the reward ([42]). Moreover, balancing between $\beta$ and $\gamma$ to achieve the desired behavior is non-trivial. For example, while all three examples shown in Table 1 get a high toxicity reward, the first two get a low total reward compared to the last one, even though they are grammatically valid sentences, since they are assigned a low likelihood by $p_{\text{ref}}$. If we set a much smaller $\beta$ to increase the weight of the toxicity reward $R_1(\mathbf{x})$, the policy $p_{\theta }$ would likely generate prompts from potentially spurious modes of the toxicity classifier, which will have high perplexity under the reference model. On the other hand, if we set $\gamma$ to a small value, the model would merely focus on the naturality score $R_2(\mathbf{x})$ and not generate toxic prompts.

To reduce sensitivity to the aforementioned parameters of the reward distribution, while preserving the mode coverage and ability of the training procedure to generalize to new modes, we propose an inexpensive retraining step that is applied following GFlowNet fine-tuning. This second step is akin to behavior cloning ([43, 9, 10]) in RL, where a policy is trained to imitate expert trajectories. First, we store all prompts sampled by the policy $p_{\theta }$ during GFlowNet fine-tuning in Stage 1. We expect this set to contain diverse and high-reward prompts discovered by off-policy exploration during GFlowNet fine-tuning. Subsequently, we filter the prompts in this set based on the toxicity score $R_1(\mathbf{x})$ and language model likelihood $R_2(\mathbf{x})$ being larger than some thresholds. The collected examples form an offline dataset, and the reference policy is fine-tuned again (from the same initial state as in Stage 1) to maximize log-likelihood of samples from this offline dataset. Stage 2 is very inexpensive in practice, taking under 5% of total (Stage 1 and 2) training time in our experiments (Table 4).

Task. We validate the efficacy of our method by red-teaming a variety of open source LMs with varying levels of saferty-tuning. Recall that the goal of red-teaming is to generate diverse prompts that elicit harmful responses from a target LLM $p_{\varphi }$. In our experiments, we use GPT-2 ([11]), Dolly-v2-7b ([12]), Llama-2-7b-chat ([14]), Llama-3.1-8B-Instruct ([15]), and Gemma-2b-it ([13]) as target LLMs. Note that we only require black-box access, i.e., we can only query a prompt and get a text response.

Evaluation. To measure effectiveness of each method, we compute the toxicity rate which is the percentage of generated prompts that are toxic. We consider a prompt toxic if the toxicity classifier $p_{\psi }$ assigns a score greater than 0.5 to an input pair consisting of the prompt $\mathbf{x}$ and response $\mathbf{y}$ generated by the target LLM. For $p_{\psi }$, we use a RoBERTa hate speech classifier ([44]) for GPT-2 and dolly-v2-7b, and Llama-Guard ([45]) for Llama-2-7b-chat, Llama-3.1-8B-Instruct, and Gemma-2b-it. To measure the diversity of the generated prompts, we embed the generated prompts using MiniLMv2 ([46]) encoder and compute the average pairwise cosine distance between embeddings of the prompts.

Scaling to a larger attacker LM. Table 2 shows the effect of scaling GFlowNet+MLE with larger and stronger attackers like Llama-3.2-1B ([15]). Scaling to a larger attacker results in significant improvements in both the toxicity rate and diversity.

GFlowNet attacks are more transferable across target LLMs. A potential advantage of generating diverse attack prompts is that prompts generated for red-teaming a given target LLM can potentially transfer to other LLMs, since some of the failure modes of a target LLM might be shared by other models, for instance, due to using similar web-filtered data or similar safety alignment recipes. To study this empirically, we train an attacker policy $p_{\theta }$ for red-teaming the Gemma-2b-it as the target LLM. We then sample $1,024$ prompts from the trained attacker LM and evaluate the number of prompts which transfer to other LLMs, i.e., elicit toxic responses from unseen LLMs: Llama-2-7b-chat, Llama-2-13b-chat, Llama-2-70b-chat, Llama-3-8b-instruct ([15]), Llama-3-70b-instruct, Gemma-7b-it, Gemma-1.1-2b-it, Gemma-1.1-7b-it, Mistral-7b-instruct-v0.2 ([17]), and Starling-7b-beta ([16]). As shown in Table 3, we find that many prompts generated by GFlowNet + MLE transfer to unseen target LLMs, outperforming all other methods across all the target LLMs except Mistral-7b-instruct-v0.2. REINFORCE generates almost identical prompts, tailored to the Gemma-2b-it target it was trained with, which consequently do not transfer to other target LLMs. This highlights a drawback of methods which do not generate diverse attacks. On the other extreme, PPO + Novelty is unable to discover any prompt that is effective in eliciting toxic responses and consequently none of the prompts transfer to any other LLM. These results further highlight the efficacy and usefulness of GFlowNet + MLE, which can generate both diverse and effective red-teaming prompts that can be transferred to red-team other LLMs. Additionally, we perform another transfer experiment targeted for a proprietary model, GPT-4o. We generate 1,024 prompts with an attacker LM trained to red-team Llama-2-7b-chat and evaluate how many prompts can elicit harmful responses from GPT-4o. On average, across five different sets of 1,024 prompts, 65% of them can successfully attack GPT-4o.

Setup. To demonstrate that attack prompts generated by red-teaming methods can enhance the safety alignment of target LLMs, we fine-tune Gemma-2b-it target LLM with the attack prompts discovered by various methods and evaluate its robustness. Specifically, we generate 1,024 prompts with each red-teaming method and generate refusal responses to the attack prompts with the target LLM using the prompt: "Refuse answering to the following instruction. attack_prompt". Subsequently, the target model is fine-tuned with LoRA ([52]) to maximize the conditional log-likelihood of the refusal responses to the attack prompts, resulting in six different fine-tuned target LLMs corresponding to each red-teaming method. Finally, each fine-tuned model generates responses to the attack prompts generated by each red-teaming method, and we measure the toxicity rate of the responses with Llama-Guard as the toxicity classifier $p_{\psi }$.

Our approach is not limited to text tokens and future work can explore the applicability to red-team multimodal models (e.g., text-to-image models ([53, 54])). Further, an interesting area of future work is extending the approach to the jailbreaking setting, where an attacker language model generates a suffix for an adversarial query prompt. Finally, in addition to red-teaming, it would be interesting to apply our method to generate prompts which can improve model performance on different tasks ([55]).

Limitations. While our approach shows promising performance for red-teaming various target language models, the performance is still limited by the classifier used to quantify the harmfulness of a response. The true harm that an LM output causes is often subjective and depends on the social context of deployment ([2]). As with other RL-based approaches, our approach is trained online (i.e., requires iteratively sampling the current model) and, consequently, requires sampling several responses from the target LLM to compute the reward during training, which can be costly.

We use PyTorch ([56]) and the Hugging Face Transformers library ([57]) to implement our models and all the baselines. All the implementation details are described in Appendix A, and our code is available at https://github.com/GFNOrg/red-teaming.

In order to study the effect of a reward function, we replace Llama-Guard ([45]) with a RoBERTa-based hate speech classifier ([44]) during the training of GFlowNet for computing the reward $R_1(\mathbf{x})$ in Equation 2. As shown in Table 6, the RoBERTa classifier assigns high toxicity score (reward) to prompts that do not actually elicit toxic responses from the Llama-2-7b-chat target model. This leads GFlowNet to generate false positive prompts, a phenomenon known as reward hacking ([58]), where a policy trained with a proxy behaves well according to the proxy but misaligns with the true objective due to mis-specifications of the proxy ([59]). Note that reward hacking is common in many RL applications ([60, 61, 62, 63]), and both PPO + Novelty and REINFORCE also suffer from the same reward hacking issue when red-teaming Gemma-2b-it and Llama-2-7b-chat models with the RoBERTa classifier. The reward hacking issue can be mitigated if we use Llama-Guard as a toxicity classifier as shown in Table 13 and Table 12. GFlowNet + MLE effectively generate prompts that elicit toxic responses from target language models. This is the reason why we use Llama-Guard for red-teaming and evaluating all the target models trained with safety alignment.

As discussed in Section 4.3, we fine-tune Gemma-2b-it target LLM with LoRA ([52]) to maximize the log-likelihood of refusal responses to the red-teaming prompts that our GFlowNet + MLE generated. Subsequently, we evaluate the safety-tuned model on Open LLM Leaderboard benchmark which consists of six datasets — ARC ([64]), HellaSwag ([65]), TruthfulQA ([66]), MMLU ([67]), and GSM8k ([68]). As shown in Table 7, there is no significant performance drop after safety-tuning, which indicates that the safety-tuned target LLM still retrain instruction following capabilities.

[1] Peter Lee (2016). Learning from Tay's introduction. https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/.

[2] Laura Weidinger et al. (2021). Ethical and social risks of harm from Language Models. arXiv preprint arXiv:2112.04359.

[3] Wei et al. (2023). Jailbroken: How does LLM safety training fail?. Neural Information Processing Systems (NeurIPS).

[4] Perez et al. (2022). Red Teaming Language Models with Language Models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pp. 3419–3448. doi:10.18653/v1/2022.emnlp-main.225. https://aclanthology.org/2022.emnlp-main.225.

[5] Zhang-Wei Hong et al. (2024). Curiosity-driven Red-teaming for Large Language Models. International Conference on Learning Representations (ICLR).

[6] Zou et al. (2023). Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.

[7] Zhao et al. (2024). Accelerating Greedy Coordinate Gradient via Probe Sampling. arXiv preprint arXiv:2403.01251.

[8] Hu et al. (2024). Amortizing intractable inference in large language models. International Conference on Learning Representations (ICLR).

[9] Scott Emmons et al. (2022). RvS: What is Essential for Offline RL via Supervised Learning?. International Conference on Learning Representations (ICLR).

[10] Eric Jang et al. (2021). BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning. Conference on Robot Learning (CoRL).

[11] Alec Radford et al. (2019). Language Models are Unsupervised Multitask Learners.

[12] Mike Conover et al. (2023). Free Dolly: Introducing the World's First Truly Open Instruction-Tuned LLM. https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm.

[13] Gemma Team Thomas Mesnard et al. (2024). Gemma: Open Models Based on Gemini Research and Technology. arXiv preprint arXiv:2403.08295.

[14] Hugo Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint arXiv:2307.09288.

[15] Dubey et al. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

[16] Zhu et al. (2023). Starling-7B: Improving LLM Helpfulness & Harmlessness with RLAIF. .

[17] Jiang et al. (2023). Mistral 7B. arXiv preprint arXiv:2310.06825.

[18] Yuntao Bai et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv preprint arXiv:2204.05862.

[19] Bai et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv preprint arXiv:2212.08073.

[20] Dinan et al. (2019). Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 4537–4546. doi:10.18653/v1/D19-1461. https://aclanthology.org/D19-1461.

[21] Xu et al. (2021). Bot-Adversarial Dialogue for Safe Conversational Agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 2950–2968. doi:10.18653/v1/2021.naacl-main.235. https://aclanthology.org/2021.naacl-main.235.

[22] Wallace et al. (2022). Analyzing Dynamic Adversarial Training Data in the Limit. In Findings of the Association for Computational Linguistics: ACL 2022. pp. 202–217. doi:10.18653/v1/2022.findings-acl.18. https://aclanthology.org/2022.findings-acl.18.

[23] Lee et al. (2023). Query-Efficient Black-Box Red Teaming via Bayesian Optimization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 11551–11574. doi:10.18653/v1/2023.acl-long.646. https://aclanthology.org/2023.acl-long.646.

[24] Mikayel Samvelyan et al. (2024). Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts. arXiv preprint arXiv:2402.16822.

[25] Xiaogeng Liu et al. (2024). AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. International Conference on Learning Representations (ICLR).

[26] Chao et al. (2023). Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419.

[27] Mantas Mazeika et al. (2024). HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. arXiv preprint arXiv:2402.04249.

[28] Meade et al. (2024). Universal Adversarial Triggers Are Not Universal. arXiv preprint arXiv:2404.16020.

[29] Emmanuel Bengio et al. (2021). Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation. Neural Information Processing Systems (NeurIPS).

[30] Jain et al. (2022). Biological Sequence Design with GFlowNets. International Conference on Machine Learning (ICML).

[31] David W Zhang et al. (2023). Robust Scheduling with GFlowNets. International Conference on Learning Representations (ICLR).

[32] Dinghuai Zhang et al. (2023). Let the flows tell: Solving graph combinatorial problems with GFlowNets. Neural Infromation Processing Systems (NeurIPS ).

[33] Deleu et al. (2022). Bayesian Structure Learning with Generative Flow Networks. Uncertainty in Artificial Intelligence (UAI).

[34] Hu et al. (2023). GFlowNet-EM for learning compositional latent variable models. International Conference on Machine Learning (ICML).

[35] van Krieken et al. (2023). A-NeSI: A scalable approximate method for probabilistic neurosymbolic inference. Neural Information Processing Systems (NeurIPS).

[36] Wei et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Neural Information Processing Systems (NeurIPS).

[37] Neel Jain et al. (2023). Baseline Defenses for Adversarial Attacks Against Aligned Language Models. arXiv preprint arXiv:2309.00614.

[38] Schulman et al. (2017). Equivalence between policy gradients and soft Q-learning. arXiv preprint arXiv:1704.06440.

[39] Bengio et al. (2023). GFlowNet foundations. Journal of Machine Learning Research. 24(210). pp. 1–55.

[40] Tuomas Haarnoja et al. (2017). Reinforcement Learning with Deep Energy-Based Policies. International Conference on Machine Learning (ICML).

[41] Malkin et al. (2022). Trajectory balance: Improved credit assignment in GFlowNets. Neural Information Processing Systems (NeurIPS).

[42] Lau et al. (2024). QGFN: Controllable Greediness with Action Values. arXiv preprint arXiv:2402.05234.

[43] Lili Chen et al. (2021). Decision Transformer: Reinforcement Learning via Sequence Modeling. Neural Information Processing Systems (NeurIPS).

[44] Vidgen et al. (2021). Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 1667–1682. doi:10.18653/v1/2021.acl-long.132. https://aclanthology.org/2021.acl-long.132.

[45] Inan et al. (2023). Llama guard: LLM-based input-output safeguard for human-AI conversations. arXiv preprint arXiv:2312.06674.

[46] Wang et al. (2021). MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pp. 2140–2151. doi:10.18653/v1/2021.findings-acl.188. https://aclanthology.org/2021.findings-acl.188.

[47] Federico Bianchi et al. (2024). Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions. International Conference on Learning Representations (ICLR).

[48] Tom B. Brown et al. (2020). Language Models are Few-Shot Learners. Neural Information Processing Systems (NeurIPS).

[49] Williams, Ronald J (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning. 8. pp. 229–256.

[50] Schulman et al. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.

[51] Zhu et al. (2018). Texygen: A Benchmarking Platform for Text Generation Models. SIGIR.

[52] Edward J Hu et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations (ICLR).

[53] Ramesh et al. (2021). Zero-shot text-to-image generation. International Conference on Machine Learning (ICML).

[54] Saharia et al. (2022). Photorealistic text-to-image diffusion models with deep language understanding. Neural Information Processing Systems (NeurIPS).

[55] Lin et al. (2023). The unlocking spell on base LLMs: Rethinking alignment via in-context learning. arXiv preprint arXiv:2312.01552.

[56] Paszke et al. (2019). Pytorch: An imperative style, high-performance deep learning library. Neural Information Processing Systems (NeurIPS).

[57] Wolfe et al. (2022). Supporting Mouthing in Signed Languages: New innovations and a proposal for future corpus building. In Proceedings of the 7th International Workshop on Sign Language Translation and Avatar Technology: The Junction of the Visual and the Textual: Challenges and Perspectives. pp. 125–130. https://aclanthology.org/2022.sltat-1.19.

[58] Skalse et al. (2022). Defining and characterizing reward gaming. Neural Information Processing Systems (NeurIPS).

[59] Alexander Pan et al. (2022). The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models. International Conference on Learning Representations (ICLR).

[60] Romain Paulus et al. (2018). A Deep Reinforced Model for Abstractive Summarization. International Conference on Learning Representations (ICLR).

[61] Yizhong Wang et al. (2023). How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources. Neural Information Processing Systems (NeurIPS).

[62] Tom Everitt et al. (2017). Reinforcement Learning with a Corrupted Reward Channel. International Joint Conference on Artificial Intelligence (IJCAI).

[63] Bowen Baker et al. (2020). Emergent Tool Use From Multi-Agent Autocurricula. International Conference on Learning Representations (ICLR).

[64] Peter Clark et al. (2018). Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv preprint arXiv:1803.05457.

[65] Zellers et al. (2019). HellaSwag: Can a Machine Really Finish Your Sentence?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 4791–4800. doi:10.18653/v1/P19-1472. https://aclanthology.org/P19-1472.

[66] Lin et al. (2022). TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 3214–3252. doi:10.18653/v1/2022.acl-long.229. https://aclanthology.org/2022.acl-long.229.

[67] Dan Hendrycks et al. (2021). Measuring Massive Multitask Language Understanding. International Conference on Learning Representations (ICLR).

[68] Cobbe et al. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.