\bf Generative Augmented Flow Networks

The Generative Flow Network ([1], GFlowNet) is a probabilistic framework where an agent learns a stochastic policy for object generation, such that the probability of generating an object is proportional to a given reward function. Its effectiveness has been shown in discovering high-quality and diverse solutions, compared to reward-maximizing reinforcement learning-based methods. Nonetheless, GFlowNets only learn from rewards of the terminal states, which can limit its applicability. Indeed, intermediate rewards play a critical role in learning, for example from intrinsic motivation to provide intermediate feedback even in particularly challenging sparse reward tasks. Inspired by this, we propose Generative Augmented Flow Networks (GAFlowNets), a novel learning framework to incorporate intermediate rewards into GFlowNets. We specify intermediate rewards by intrinsic motivation to tackle the exploration problem in sparse reward environments. GAFlowNets can leverage edge-based and state-based intrinsic rewards in a joint way to improve exploration. Based on extensive experiments on the GridWorld task, we demonstrate the effectiveness and efficiency of GAFlowNet in terms of convergence, performance, and diversity of solutions. We further show that GAFlowNet is scalable to a more complex and large-scale molecule generation domain, where it achieves consistent and significant performance improvement.

Deep reinforcement learning (RL) has achieved significant progress in recent years with particular success in games ([2, 3, 4]). RL methods applied to the setting where a reward is only given at the end (i.e., terminal states) typically aim at maximizing that reward function for learning the optimal policy. However, diversity of the generated states is desirable in a wide range of practical scenarios including molecule generation ([5]), biological sequence design ([6]), recommender systems ([7]), dialogue systems ([8]), etc. For example, in molecule generation, the reward function used in in-silico simulations can be uncertain and imperfect itself (compared to the more expensive in-vivo experiments). Therefore, it is not sufficient to only search the solution that maximizes the return. Instead, it is desired that we sample many high-reward candidates, which can be achieved by sampling them proportionally to the reward of each terminal state.

Interestingly, GFlowNets ([5, 1]) learn a stochastic policy to sample composite objects $\mathbf{x}\in \mathcal{X}$ with probability proportional to the return $R(\mathbf{x})$. The learning paradigm of GFlowNets is different from other RL methods, as it is explicitly aiming at modeling the diversity in the target distribution, i.e., all the modes of the reward function. This makes it natural for practical applications where the model should discover objects that are both interesting and diverse, which is a focus of previous GFlowNet works ([5, 1, 9, 6]).

Yet, GFlowNets only learn from the reward of the terminal state, and do not consider intermediate rewards, which can limit its applicability, especially in more general RL settings. Rewards play a critical role in learning ([10]). The tremendous success of RL largely depends on the reward signals that provide intermediate feedback. Even in environments with sparse rewards, RL agents can motivate themselves for efficient exploration by intrinsic motivation, which augments the sparse extrinsic learning signal with a dense intrinsic reward at each step. Our focus in this paper is precisely on introducing such intermediate intrinsic rewards in GFlowNets, since they can be applied even in settings where the extrinsic reward is sparse (say non-zero only on a few terminal states).

Inspired by this missing element of GFlowNets, we propose a new GFlowNet learning framework that takes intermediate feedback signals into account to provide an exploration incentive during training. The notion of flow in GFlowNets ([5, 1]) refers to a marginalized quantity that sums rewards over all downstream terminal states following a given state, while sharing that reward with other states leading to the same terminal states. Apart from the existing flows in the network, we introduce augmented flows as intermediate rewards. Our new framework is well-suited for sparse reward tasks by considering intrinsic motivation as intermediate rewards, where the training of GFlowNet can get trapped in a few modes, since it may be difficult for it to discover new modes based on those it visited ([1]).

We first propose an edge-based augmented flow, based on the incorporation of an intrinsic reward at each transition. However, we find that although it improves learning efficiency, it only performs local exploration and still lacks sufficient exploration ability to drive the agent to visit solutions with zero rewards. On the other hand, we find that incorporating intermediate rewards in a state-based manner ([1]) can result in slower convergence and large bias empirically, although it can explore more broadly. Therefore, we propose a joint way to take both edge-based and state-based augmented flows into account. Our method can improve the diversity of solutions and learning efficiency by reaping the best from both worlds. Extensive experiments on the GridWorld and molecule domains that are already used to benchmark GFlowNets corroborate the effectiveness of our proposed framework.

Consider a directed acyclic graph (DAG) $G=(\mathcal{S},\mathbb{A})$, where $\mathcal{S}$ denotes the state space, and $\mathbb{A}$ represents the action space, which is a subset of $\mathcal{S}\times \mathcal{S}$. We denote the vertex $s_0\in \mathcal{S}$ to be the initial state with no incoming edges, while the vertex $s_f$ without outgoing edges is called the sink state, and state-action pairs correspond to edges. The goal for GFlowNets is to learn a stochastic policy $\pi$ that can construct discrete objects $\mathbf{x}\in \mathcal{X}$ with probability proportional to the reward function $R:\mathcal{X}\to {\mathbb{R}}_{\ge 0}$, i.e., $\pi (\mathbf{x})\propto R(\mathbf{x})$. GFlowNets construct objects sequentially, where each step adds an element to the construction. We call the resulting sequence of state transitions from the initial state to a terminal state $\tau =({\mathbf{s}}_0\to \cdots \to {\mathbf{s}}_n)$ a trajectory, where $\tau \in \mathcal{T}$ with $\mathcal{T}$ denoting the set of trajectories. [5] define a trajectory flow $F:\mathcal{T}\to {\mathbb{R}}_{\ge 0}$. Let $F(\mathbf{s})={\sum }_{\tau \ni \mathbf{s}}F(\tau )$ define a state flow for any state $\mathbf{s}$, and $F(\mathbf{s}\to {\mathbf{s}}^{\prime })={\sum }_{\tau \ni \mathbf{s}\to {\mathbf{s}}^{\prime }}F(\tau )$ defines the edge flow for any edge $\mathbf{s}\to {\mathbf{s}}^{\prime }$. The trajectory flow induces a probability measure $P(\tau )=\frac{F(\tau )}{Z}$, where $Z={\sum }_{\tau \in \mathcal{T}}F(\tau )$ denotes the total flow. We then define the corresponding forward policy $P_F({\mathbf{s}}^{\prime }|\mathbf{s})=\frac{F(\mathbf{s}\to {\mathbf{s}}^{\prime })}{F(\mathbf{s})}$ and the backward policy $P_B(\mathbf{s}|{\mathbf{s}}^{\prime })=\frac{F(\mathbf{s}\to {\mathbf{s}}^{\prime })}{F({\mathbf{s}}^{\prime })}$. The flows can be considered as the amount of water flowing through edges (like pipes) or states (like tees connecting pipes) ([9]), with $R(\mathbf{x})$ the amount of water through terminal state $\mathbf{x}$, and $P_F({\mathbf{s}}^{\prime }|\mathbf{s})$ the relative amount of water flowing in edges outgoing from $\mathbf{s}$.

We call a flow consistent if it satisfies the flow matching constraint for all internal states $\mathbf{s}$, i.e., ${\sum }_{{\mathbf{s}}^{\prime \prime }\to \mathbf{s}}F({\mathbf{s}}^{\prime \prime }\to \mathbf{s})=F(\mathbf{s})={\sum }_{\mathbf{s}\to {\mathbf{s}}^{\prime }}F(\mathbf{s}\to {\mathbf{s}}^{\prime })$, which means that the incoming flows equal the outgoing flows. [5] prove that for a consistent flow $F$ where the terminal flow is set to be the reward, the forward policy can sample objects $x$ with probability proportional to $R(\mathbf{x})$.

Flow matching (FM). [5] propose to approximate the edge flow by a model $F_{\theta }(\mathbf{s},{\mathbf{s}}^{\prime })$ parameterized by $\theta$ following the FM objective, i.e., ${\mathcal{L}}_{FM}(\mathbf{s})=(\log {\sum }_{({\mathbf{s}}^{\prime \prime }\to \mathbf{s})\in \mathcal{A}}{F}_{\theta }({\mathbf{s}}^{\prime \prime },\mathbf{s})-\log {\sum }_{(\mathbf{s}\to {\mathbf{s}}^{\prime })\in \mathcal{A}}{F}_{\theta }(\mathbf{s},{\mathbf{s}}^{\prime }){)}^2$ for non-terminal states. At terminal states, a similar objective encourages the incoming flow to match the corresponding reward. The objective is optimized using trajectories sampled from a training policy $\pi$ with full support such as a tempered version of $P_{F_{\theta }}$ or a mixture of $P_{F_{\theta }}$ with a uniform policy $U$, i.e., ${\pi }_{\theta }=(1-ϵ)P_{F_{\theta }}+ϵ\cdot U$, This is similar to $ϵ$-greedy and entropy-regularized strategies in RL to improve exploration. [5] prove that if we reach a global minimum of the expected loss function and the training policy ${\pi }_{\theta }$ has full support, then GFlowNet samples from the target distribution.

Detailed balance (DB). [1] propose the DB objective to avoid the computationally expensive summing operation over the parents or children of states. For learning based on DB, we train a neural network with a state flow model $F_{\theta }$, a forward policy model $P_{F_{\theta }}(\cdot |\mathbf{s})$, and a backward policy model $P_{B_{\theta }}(\cdot |\mathbf{s})$ parameterized by $\theta$. The optimization objective is to minimize ${\mathcal{L}}_{DB}(\mathbf{s},{\mathbf{s}}^{\prime })={\left(\log (F_{\theta }(\mathbf{s})P_{F_{\theta }}({\mathbf{s}}^{\prime }|\mathbf{s}))-\log (F_{\theta }({\mathbf{s}}^{\prime })P_{B_{\theta }}(\mathbf{s}|{\mathbf{s}}^{\prime }))\right)}^2$. It also samples from the target distribution if a global minimum of the expected loss is reached and ${\pi }_{\theta }$ has full support.

Trajectory balance (TB). [9] propose the TB objective for faster credit assignment and learning over longer trajectories. The loss function for TB is ${\mathcal{L}}_{TB}(\tau )=(\log (Z_{\theta }{\prod }_{t=0}^{n-1}{P}_{F_{\theta }}({\mathbf{s}}_{t+1}|{\mathbf{s}}_t))-\log (R(\mathbf{x}){\prod }_{t=0}^{n-1}{P}_B({\mathbf{s}}_t|{\mathbf{s}}_{t+1})){)}^2$, where $Z_{\theta }$ is a learnable parameter.

GFlowNets. Since the proposal of GFlowNets ([5]), there has been an increasing interest in improving ([1, 9]), understanding, and applying this framework to practical scenarios. It is a general-purpose high-level probabilistic inference framework, and induces fruitful applications ([11, 12, 13, 14]). However, previous works only consider learning based on the terminal reward, which can make it difficult to provide a good training signal for intermediate states, especially when the reward is sparse (i.e., significantly non-zero in only a tiny fraction of the terminal states).

Reinforcement learning (RL). Different from GFlowNets that aim to sample proportionally to the reward function, RL learns a reward-maximization policy. Although introducing entropy regularization to RL ([15, 16, 17, 18]) can improve diversity, this is limited to tree-structured DAGs. This is because it could only sample a terminal state $\mathbf{x}$ in proportion to the sum of rewards over all trajectories leading to $\mathbf{x}$. It can fail on general (non-tree) DAGs ([5]) for which the same terminal state $\mathbf{x}$ can be obtained with a potentially large number of trajectories (and a very different number of trajectories for different $\mathbf{x}$'s).

Intrinsic motivation. There has been a line of research to incorporate intrinsic motivation ([19, 20, 21]) for improving exploration in RL. Yet, such ideas have not been explored with GFlowNets because the current mathematical framework of GFlowNets only allows for terminal rewards, unlike the standard RL frameworks. This deficiency as well as the potential of introducing intrinsic intermediate rewards motivates this paper.

The potential difficulty in learning only from the terminal reward is related to the challenge of sparse rewards in RL, where most states do not provide an informative reward. We demonstrate the sparse reward problem for GFlowNets and reveal interesting findings based on the GridWorld task (as shown in Figure 4) with sparse rewards. Specifically, the agent only receives a reward of $+1$ only when it reaches one of the $3$ goals located around the corners of the world (except the starting state corner) with size $H\times H$ (with $H\in \{64,128\}$), and the reward is $0$ otherwise. A more detailed description of the task can be found in Section 5.1. We evaluate the number of modes discovered by the GFlowNet trained with TB, following [5]. As summarized in Figure 1, GFlowNet training can get trapped in a subset of the modes. Therefore, it remains a critical challenge for GFlowNets to efficiently learn when the reward signal is sparse and non-informative.

On the other hand, there has been recent progress with intrinsic motivation methods ([19, 20]) to improve exploration of RL algorithms, where the agent learns from both a sparse extrinsic reward and a dense intrinsic bonus at each step. Building on this, we aim to address the exploration challenge of GFlowNets by enabling intermediate rewards in GFlowNets and thus intrinsic rewards.

In contrast to simply adding a constant uniform probability to every action (which is commonly used for exploration with GFlowNets), the pseudo-exit intermediate rewards have an effect that is not local. In addition, we can specify non-uniform intermediate rewards as intrinsic motivation $r({\mathbf{s}}_t\to {\mathbf{s}}_{t+1})$ with novelty-based methods ([19, 20]) to better tackle exploration in sparse reward tasks. Although how to accurately measure the novelty degree remains an open problem, it has been shown that random network distillation (RND) ([20]) is a simple yet effective method for encouraging the agent to visit states of interest. We use the difference between the predicted features by a trainable state encoder and a random fixed state encoder as the intrinsic rewards based on RND, i.e., $||\varphi (\mathbf{s})-{\bar{\varphi }}_{random}(\mathbf{s})|{|}_2$. The random distillation network is trained by minimizing such differences. Therefore, the novelty measure is generally smaller for more often seen states or similar states. The overall training procedure is shown in Algorithm 1 by substituting the augmented trajectory balance loss according to Eq. (4).

Following the evaluation scheme in ([5]), we summarize the number of discovered modes and the empirical $L_1$ error for GFlowNet and GAFlowNet with edge-based reward augmentation in Figure 3. The figure also includes the state-based and joint objectives introduced in later sections (Section 4.2 and Section 4.3) for completeness. The $L_1$ error is defined as $\mathbb{E}\left[|p(\mathbf{x})-\pi (\mathbf{x})|\right]$, where $p(\mathbf{x})=R(\mathbf{x})/Z$ denotes the true reward distribution, and we estimate $\pi$ by repeated sampling and summarizing frequencies for visitation of each possible state $\mathbf{x}$. As shown, GAFlowNet (edge) is able to discover more modes as opposed to a standard GFlowNet and improves diversity. In addition, it learns more efficiently and leads to a smaller level of $L_1$ error.

Different from the edge-based reward augmentation, [1] defines a trajectory return as the sum of intermediate rewards in a state-based reward augmentation manner. Specifically, state-based reward augmentation for trajectory balance yields the following criterion

In Theorem 1, we theoretically justify that the resulting joint augmentation method leads to an unbiased solution to the original formulation asymptotically. The proof can be found in Appendix A. Note that we employ RND ([20]) for the intrinsic rewards, which decrease as the agent has more knowledge about the state.

We first conduct a series of experiments based on GridWorld with sparse rewards (Figure 4). The task is the same as introduced in ([5]), except that the reward function is sparse as described in Section 4, which makes it much harder due to the challenge of exploration. With a larger value of the size $H$, it requires the agent to plan in a longer horizon and learn from sparse reward signals. Actions include operations to increase one coordinate as in ([5]), and a stop operation indicating termination to guarantee that the underlying MDP is a directed acyclic graph. We compare GAFlowNet against strong baselines including Metropolis-Hastings-MCMC ([22]), PPO ([23]), and a GFlowNet ([9]). We also include a variant of PPO with intrinsic rewards based on the same intrinsic motivation mechanism using RND ([20]). All baselines are implemented based on the open-source code¹. Each algorithm is run for five random seeds, and we report their mean and standard deviation. A detailed description of the hyperparameters and setup can be found in Appendix B.1.

We now demonstrate that our proposed framework is versatile by building it upon the other two GFlowNet objectives based on the detailed balance (DB) ([1]) and flow matching (FM) ([5]) criteria. Comparison of empirical $L_1$ error averaged over increasing sizes $H$ are summarized in Figure 7. As demonstrated, GAFlowNet also significantly improves training convergence of DB and FM, which provides consistent improvement gains.

We now investigate the effectiveness of our method in larger-scale tasks, by evaluating it on the more challenging molecule generation task ([5]) as depicted in Figure 8(a). A molecule is represented by a graph, which consists of a vocabulary of building blocks. The agent sequentially generates the molecule by choosing where to attach a block and also which block to attach at each step considering chemical validity constraints. There is also an exit action indicating whether the agent decides to stop the generation process. This problem is challenging with large state (about $10^{16}$) and action (around $100$ to $2000$) spaces. The agent aims to discover diverse molecules with high rewards, i.e., low binding energy to the soluble epoxide hydrolase (sEH) protein. We use a pretrained proxy model to compute this binding energy. We consider a sparse reward function here, where the agent only obtains a non-zero reward if the corresponding molecule succeeds to meet a target score, and the reward is $0$ otherwise. A detailed description of the environment is in Appendix B.1.1. We compare our method with previous GFlowNet results ([5]), PPO ([23]), PPO with intrinsic rewards based on RND, and MARS ([24]). All baselines are implemented based on the open-source code² and run with three random seeds as in ([5]). More details for the setup can be found in Appendix B.1.2.

We follow the evaluation metric in ([5]) and investigate our method in both performance and diversity. Figure 8(b) demonstrates the average reward of the top-$10$ unique molecules generated by each method. The number of modes discovered by each method with rewards above $7.5$ is summarized in Figure 8(c). We compute the average pairwise Tanimoto similarities for the top-$10$ samples in Figure 8(d). Additional comparison results can be found in Appendix B.5.

The molecule generation task We adopt a pretrained proxy model for the reward, which is trained on a dataset of $300,000$ molecules that are randomly generated as provided in ([5]). For the original dense reward function, the agent receives a reward based on the normalized score. Here, we use a sparse reward function, where the agent only obtains the original non-zero reward if the normalized score succeeds to meet a target score ($7.0$), and the reward is $0$ otherwise. As described in Section 5.2.1, the agent can choose one of the blocks to attach from the basic building blocks vocabulary (with a size of $105$).

All baseline methods are implemented based on the open-source implementation as described in the main text, where we follow the default hyperparameters and setup as in ([5]). Specifically, in GridWorld, the GFlowNet model is a feedforward network consisting of two hidden layers with $256$ hidden units per layer using LeakyReLU activation. We train all models based on samples from a parallel of $16$ rollouts in the environment. We leverage random network distillation (RND) ([20]) as the intrinsic reward mechanism, where the random target network and the predictor network are both feedforward networks consisting of two hidden layers with $256$ hidden units per layer using LeakyReLU activation. We train the GFlowNet model and RND jointly based on the Adam ([25]) optimizer with a learning rate of $0.001$ for the policy models ($P_F$ and $P_B$) and $0.1$ for $Z$. For the molecule generation task, we use a reward proxy provided in ([5]). As the molecule is represented as an atom graph, we use Message Passing Neural Networks (MPNN) ([26]) as the network architecture for all models. Note that we build our method upon GFlowNet based on the flow matching criterion in the molecule generation task, since it is the most competitive version in this task in terms of finding high-quality and diverse candidates. For GAFlowNet, the only hyperparameter that requires tuning is the coefficient $\alpha$ of intrinsic rewards, where we use a same value for state-based and edge-based augmentation. We tune $\alpha$ in $\{0.001,0.005,0.01,0.05,0.1,0.5\}$ with grid search. Specifically, $\alpha =0.001$ for GridWorld with all values of horizon except for $H=64$, where we set $\alpha$ to be $0.005$. For the molecule generation task, $\alpha$ is set to be $0.1$. The code will be released upon publication of the paper.

Apart from evaluating our method based on the metrics (the number of modes discovered by each method and empirical $L_1$ error) as in ([5]), we are also interested in its performance after each update. Here, we evaluate the performance of baselines after each update (instead of throughout the training process) as in the evaluation scheme of RL algorithms. Figure 11 demonstrates the performance for the top-$5$ solutions among a batch of $16$ parallel rollouts of each method after each update for GridWorld with sizes $H\in \{8,16,32,64,128\}$. As shown, although PPO is more efficient than PPO, both of them underperform GFlowNet by a large margin (especially with a larger value of $H$). We find that GAFlowNet significantly outperforms baseline methods, and also performs more efficiently than GFlowNet.

We investigate the effect of different types of intrinsic rewards including Intrinsic Curiosity Module (ICM) ([19]), Novelty Difference (NovelD) ([21]), and Random Network Distillation (RND) ([20]) in Figure 12, with fine-tuned coefficients for intrinsic rewards. We also include a baseline with constant intrinsic rewards in GAFlowNet, which mimics the behavior of $ϵ$-greedy exploration typically used in reinforcement learning algorithms. As demonstrated, GAFlowNet is not sensitive to the forms of intrinsic rewards, but RND enables the fastest convergence in our simulations. It also validates the effectiveness of the novelty-based methods from the comparison of GAFlowNet and GAFlowNet with constant intrinsic rewards. This is because novelty-based methods are more efficient than blindly wandering in the maze.

Following the evaluation metrics in ([5]), besides the results in Figure 8 in the main text, we also evaluate the average reward of the top-$100$ molecules and the number of modes with $R>8.0$ discovered by each method. As demonstrated in Figure 13, GAFlowNet achieves consistent and significant performance improvement over previous baselines.

[1] Yoshua Bengio, Tristan Deleu, Edward J Hu, Salem Lahlou, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. arXiv preprint arXiv:2111.09266, 2021b.

[2] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.

[3] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016.

[4] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019.

[5] Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381–27394, 2021a.

[6] Moksh Jain, Emmanuel Bengio, Alex Hernandez-Garcia, Jarrid Rector-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference on Machine Learning, pages 9786–9801. PMLR, 2022b.

[7] Matevž Kunaver and Tomaž Požrl. Diversity in recommender systems–a survey. Knowledge-based systems, 123:154–162, 2017.

[8] Yichi Zhang, Zhijian Ou, and Zhou Yu. Task-oriented dialog systems that consider multiple appropriate responses under the same context. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9604–9611, 2020.

[9] Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in gflownets. arXiv preprint arXiv:2201.13259, 2022.

[10] David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. Reward is enough. Artificial Intelligence, 299:103535, 2021.

[11] Dinghuai Zhang, Ricky T. Q. Chen, Nikolay Malkin, and Yoshua Bengio. Unifying generative models with gflownets. 2022a.

[12] Dinghuai Zhang, Nikolay Malkin, Z. Liu, Alexandra Volokhova, Aaron C. Courville, and Yoshua Bengio. Generative flow networks for discrete probabilistic modeling. In ICML, 2022b.

[13] Tristan Deleu, Ant'onio G'ois, Chris C. Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian structure learning with generative flow networks. ArXiv, abs/2202.13903, 2022.

[14] Moksh Jain, Emmanuel Bengio, Alex García, Jarrid Rector-Brooks, Bonaventure F. P. Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Micheal Kilgour, Dinghuai Zhang, Lena Simine, Payel Das, and Yoshua Bengio. Biological sequence design with gflownets. In ICML, 2022a.

[15] Hagai Attias. Planning by probabilistic inference. In AISTATS, 2003.

[16] Brian D. Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010.

[17] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pages 1352–1361. PMLR, 2017.

[18] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. PMLR, 2018.

[19] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778–2787. PMLR, 2017.

[20] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018.

[21] Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian. Noveld: A simple yet effective exploration criterion. Advances in Neural Information Processing Systems, 34:25217–25230, 2021.

[22] Hanjun Dai, Rishabh Singh, Bo Dai, Charles Sutton, and Dale Schuurmans. Learning discrete energy-based models via auxiliary-variable local exploration. Advances in Neural Information Processing Systems, 33:10443–10455, 2020.

[23] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.

[24] Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. Mars: Markov molecular sampling for multi-objective drug discovery. In International Conference on Learning Representations, 2020.

[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.

[26] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263–1272. PMLR, 2017.