Dyna-Mind: Learning to Simulate from Experience for Better AI Agents

Xiao YuBaolin PengMichel GalleyHao ChengQianhui WuJanardhan KulkarniSuman NathZhou YuJianfeng Gao

article2025arXiv8 citations

Introduces Dyna-Mind, a training framework combining experience-grounded simulation reasoning with online reinforcement learning to equip AI agents with the ability to mentally evaluate future states before acting in complex, long-horizon interactive environments.

Listen

Artificial intelligence reasoning models have achieved strong results in static domains like mathematics and coding, but they frequently struggle in interactive, long-horizon tasks such as operating mobile phones, navigating graphical user interfaces, and solving multi-step text puzzles. These failures occur largely because current agents lack internal world models to simulate potential future states before committing to actions. The article evaluates and demonstrates Dyna-Mind, a two-stage training framework designed to teach vision-language agents how to mentally simulate environments and use those simulations to improve multi-step decision-making.

The framework integrates world simulation into an agent's reasoning process across two training phases. In the first stage, called Reasoning with Simulations, the system builds search trees from real environment rollouts, values each branch, and condenses these explored paths into single, structured reasoning traces used for supervised fine-tuning. In the second stage, an online reinforcement learning algorithm named Dyna-GRPO alternates between standard policy improvement and simulation refinement. During refinement steps, real future state feedback is provided to prompt the agent to correct its intermediate reasoning, optimizing both final task success and world simulation accuracy. The authors evaluated the approach on two text-based environments (Sokoban and ALFWorld) and one realistic graphical benchmark (AndroidWorld).

The evaluations yielded several key findings. First, simulation accuracy strongly correlates with final task success, showing that accurate internal foresight is a key driver of agent competence. Second, in text-based environments, Dyna-Mind models achieved an average success rate of 77.1% on Sokoban and 90.8% on ALFWorld, outperforming standard reinforcement learning baselines (which achieved 73.1% and 87.0% respectively) while using up to eleven times fewer output tokens than large reasoning models like DeepSeek-R1. Third, on the complex AndroidWorld benchmark, a 32-billion parameter model trained with Dyna-Mind achieved a 31.8% average success rate (40.7% in-distribution and 22.9% out-of-distribution), outperforming both the standard prompting baseline of 19.5% and standard reinforcement learning at 27.8%.

These findings indicate that autonomous agents do not require separate, computationally heavy search modules at runtime if world dynamics are baked directly into their internalized reasoning traces. By generating concise, simulation-grounded plans, agents reduce computational latency and token costs while making fewer irreversible operational mistakes. The article recommends adopting end-to-end simulation-guided training for autonomous agents and using environment interaction rollouts to refine agent foresight during reinforcement learning.

The approach currently faces limitations in visual environments. On AndroidWorld, performance remains constrained by underlying visual model errors, including misinterpreting complex interface icons and an inability to recover after sequential interface mistakes. While confidence is high that mental simulation improves agent planning across domains, scaling these techniques to real-world software will require stronger foundational visual perception.

Columbia University Microsoft reasoning reinforcement learning method intelligent agents simulation-guided reasoning world simulation DYNA-GRPO training

arXiv: 2510.09577

Paper: Reasoning with Language Model is Planning with World Model, Shibo Hao et al. (2023). This paper establishes the foundation for repurposing language models to act as internal world models for planning over simulated states, a core prerequisite conceptualized in Dyna-Mind's simulation framework.
Paper: ReAct: Synergizing Reasoning and Acting in Language Models, Shunyu Yao et al. (2023). It provides the seminal paradigm of interleaving reasoning traces and environment actions in language models that Dyna-Mind builds upon and augments with internal simulations.
Paper: Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models, Andy Zhou et al. (2024). This work unifies reasoning, acting, and tree-search planning with environment feedback, directly preceding Dyna-Mind's search-tree experience generation (ReSim).
Paper: Dream to Control: Learning Behaviors by Latent Imagination, Danijar Hafner et al. (2019). It introduces the foundational concept of learning behaviors entirely through latent imagination in learned world models, which inspires Dyna-Mind's mental simulation mechanism.
Paper: Recurrent World Models Facilitate Policy Evolution, David Ha et al. (2018). This foundational paper demonstrates how recurrent world models enable agents to simulate futures internally for policy optimization, motivating Dyna-Mind's vicarious trial-and-error reasoning.
Paper: Reflexion: language agents with verbal reinforcement learning, Noah Shinn et al. (2023). It establishes iterative verbal self-reflection and learning from trial outcomes in interactive environments like ALFWorld, which Dyna-Mind formalizes into structured simulation training.
Paper: Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning, Thomas Carta et al. (2023). This work details grounding language models in interactive environments via online reinforcement learning, providing essential context for Dyna-Mind's second-stage RL optimization.
Paper: DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning, A list of authors and their affiliations appears at the end of the paper (2025). It introduces Group Relative Policy Optimization (GRPO) for reasoning models, which Dyna-Mind adapts into Dyna-GRPO to leverage intermediate states and outcome rewards.

Paper: Agentic Reasoning for Large Language Models, Tianxin Wei et al. (2026). This comprehensive survey organizes advanced agentic reasoning and self-evolving mechanisms, contextualizing Dyna-Mind's simulation-grounded planning within the broader agent landscape.
Paper: Experiential Reinforcement Learning, Taiwei Shi et al. (2026). It builds upon experiential reinforcement learning in interactive puzzle environments like Sokoban by internalizing structured reflections into deployment policies.
Paper: From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills, Zisu Huang et al. (2026). This paper investigates how raw interactive execution logs can be distilled into reusable agent skills across complex environments like ALFWorld, extending experience-driven reasoning.
Paper: Reasoning Models Generate Societies of Thought, Junsol Kim et al. (2026). It analyzes the internal multi-perspective dynamics of reinforcement-learned reasoning traces, providing deeper mechanistic insight into how simulated reasoning trajectories emerge.

Cover for Dyna-Mind: Learning to Simulate from Experience for Better AI Agents

Abstract

Reasoning models have recently shown remarkable progress in domains such as math and coding. However, their expert-level abilities in math and coding contrast sharply with their performance in long-horizon, interactive tasks such as web navigation and computer/phone-use. Inspired by literature on human cognition, we argue that current AI agents need ''vicarious trial and error'' - the capacity to mentally simulate alternative futures before acting - in order to enhance their understanding and performance in complex interactive environments. We introduce Dyna-Mind, a two-stage training framework that explicitly teaches (V)LM agents to integrate such simulation into their reasoning. In stage 1, we introduce Reasoning with Simulations (ReSim), which trains the agent to generate structured reasoning traces from expanded search trees built from real experience gathered through environment interactions. ReSim thus grounds the agent's reasoning in faithful world dynamics and equips it with the ability to anticipate future states in its reasoning. In stage 2, we propose Dyna-GRPO, an online reinforcement learning method to further strengthen the agent's simulation and decision-making ability by using both outcome rewards and intermediate states as feedback from real rollouts. Experiments on two synthetic benchmarks (Sokoban and ALFWorld) and one realistic benchmark (AndroidWorld) demonstrate that (1) ReSim effectively infuses simulation ability into AI agents, and (2) Dyna-GRPO leverages outcome and interaction-level signals to learn better policies for long-horizon, planning-intensive tasks. Together, these results highlight the central role of simulation in enabling AI agents to reason, plan, and act more effectively in the ever more challenging environments.

Topics

reasoning reinforcement learning method intelligent agents software agents autonomous agents

Keywords

simulation-guided reasoning world simulation DYNA-GRPO training autonomous agents task-completion dialogue policy learning planning-intensive tasks simulation refinement Dyna algorithms

1 Introduction
2 Related Work
3 Dyna-Mind
3.1 Notation
3.2 Reasoning with Simulations (ReSim)
3.3 Dyna-GRPO
4 Experiments
4.1 Text Games
4.1.1 Main Results
4.1.2 Measuring Simulation Ability
4.2 AndroidWorld
4.2.1 Main Results
5 Conclusion
References
A LLM Usage
B Ethics Statement
C Additional Algorithmic Details
D Additional Details on Text Games
D.1 Example Tasks and Actions
D.2 ReSim Implementation Details
D.3 Simulation Refinement Performance
D.4 Additional Training Details
D.5 Simulation Score Prompts
E Additional Details on AndroidWorld
E.1 Example Task and Actions in AndroidWorld
E.2 Additional Training Details
E.3 Other Implementation/Evaluation Details

Knowls

Knowl 1 — Dyna-Mind Framework for Grounded Simulation in Language Model Agents
model/method
Dyna-Mind is a two-stage training framework designed to teach (visual) language model ((V)LM) agents to mentally simulate future environment states before selecting actions during interactive, multi-step tasks.

The framework addresses the issue where (V)LM agents hallucinate environment dynamics in long-horizon planning tasks:
1. Stage 1 (Reasoning with Simulations / RESIM): Synthesizes training data by constructing expanded search trees (via depth-first search or rollouts) grounded in real environment interactions. A (V)LM aggregates these search trees—including ground-truth intermediate states and discounted value estimates—into structured reasoning traces ( $a^{\text{ReSim}}$ ). The policy model is then trained via supervised fine-tuning (SFT) to generate these simulation-guided reasoning traces directly from observation histories without requiring tree search during inference.
2. Stage 2 (Dyna-GRPO): An online reinforcement learning algorithm that alternates between simulation improvement and policy improvement. In simulation improvement, the agent generates simulation refinement rollouts (SIMROLLOUT) where proposed plans are executed to obtain true future states, prompting the model to refine its simulation. The model is trained on both unrefined trajectories and refined trajectories with a specialized advantage function. In policy improvement, standard rollouts without future-state information are optimized using Group Relative Policy Optimization (GRPO).
Knowl 2 — Reasoning with Simulations (RESIM) Search Tree Aggregation and Distillation
algorithm
Reasoning with Simulations (RESIM) generates structured imitation learning data by executing search trees in the environment and condensing the exploration into a single simulation-guided reasoning trace per step.
Input: Rollout policy $\pi_\theta$ , value function $V_\nu$ , environment $\mathcal{T}$ , aggregation model $M$ , branching factor $b$ , search depth $d$ , maximum episode steps $t_{\text{max}}$ , training branch count $b_{\text{train}}$
Output: Trajectory $\tau$ of states and simulation-guided reasoning traces
Initialize trajectory $\tau \leftarrow \{\}$ , time step $t \leftarrow 0$ , state $s_0 \sim \mathcal{T}$
while task not done and $t < t_{\text{max}}$ do
    Sample $b$ rollouts $\{\tau^i\}_{i=1}^b$ of depth up to $d$ from state $s_t$ using policy $\pi_\theta$
    Deduplicate sampled rollouts into unique set $\{\tau^i\}_{i=1}^{b'}$
    Compute terminal state value estimates $v^i = V_\nu(s_{t+d}^i)$ for each rollout $i \in \{1, \dots, b'\}$
    Identify best rollout $\tau^* = \arg\max_{\tau^i} v^i$
    Form subset $\{\tau^i\}_{i=1}^{b_{\text{train}}}$ consisting of $\tau^*$ and $b_{\text{train}} - 1$ subsampled rollouts from the remainder
    for each rollout $i \in \{1, \dots, b_{\text{train}}\}$ do
         $\text{plan}^i \leftarrow \text{summarize}(M, \tau^i, v^i)$
    end for
     $a^{\text{ReSim}} \leftarrow \text{aggregate}(M, s_t, \{\text{plan}^i\}_{i=1}^{b_{\text{train}}})$
    Execute chosen action from $a^{\text{ReSim}}$ in $\mathcal{T}$ to transition $s_{t+1} \sim \mathcal{T}(s_t, a^{\text{ReSim}})$
     $\tau \leftarrow \tau \cup \{s_t, a^{\text{ReSim}}\}$
     $t \leftarrow t + 1$
end while
return $\tau$
Following trajectory collection, the agent is trained via supervised fine-tuning (SFT) on the resulting sequence $\tau = \{s_0, a_0^{\text{ReSim}}, s_1, a_1^{\text{ReSim}}, \dots, s_T, a_T^{\text{ReSim}}\}$ to predict $a_t^{\text{ReSim}}$ given the current state $s_t$ and up to $h$ historical state-action pairs.
Knowl 3 — Simulation Refinement Rollout (SIMROLLOUT)
algorithm
SIMROLLOUT generates refined action trajectories by executing an agent's planned actions in the real environment to retrieve true intermediate states, then prompting the agent to refine its reasoning and action conditioned on these future ground-truth states.
Input: Policy $\pi_\theta$ , environment $\mathcal{T}$ , group size $G$ , plan depth $n$ , maximum episode steps $t_{\text{max}}$
Output: Refined trajectory set $\{\tau'\}$ , conditioned refined trajectory set $\{\tau'_{\text{refine}}\}$
Initialize sets $\{\tau'\} \leftarrow \{\}$ , $\{\tau'_{\text{refine}}\} \leftarrow \{\}$
repeat $G$ times
    Initialize episode buffers $\tau' \leftarrow \{\}$ , $\tau'_{\text{refine}} \leftarrow \{\}$ , $t \leftarrow 0$ , $s_0 \sim \mathcal{T}$
    while task not done and $t < t_{\text{max}}$ do
        Sample initial response $a \sim \pi_\theta(\cdot \mid s_t)$
        Extract multi-step action plan $\{\hat{a}_1, \dots, \hat{a}_n\}$ from $a$
        Execute plan in environment to obtain true future states: $s_{t+1} \sim \mathcal{T}(s_t, \hat{a}_1), \dots, s_{t+n} \sim \mathcal{T}(s_{t+n-1}, \hat{a}_n)$
        Construct refinement prompt: $s_t^{\text{refine}} = \{s_t \oplus a \oplus s_{t+1} \oplus \hat{a}_2 \oplus \dots \oplus s_{t+n}\}$
        Sample refined response $a^{\text{refine}} \sim \pi_\theta(\cdot \mid s_t^{\text{refine}})$
         $\tau' \leftarrow \tau' \cup \{s_t, a^{\text{refine}}\}$
         $\tau'_{\text{refine}} \leftarrow \tau'_{\text{refine}} \cup \{s_t^{\text{refine}}, a^{\text{refine}}\}$
        Execute $a^{\text{refine}}$ to obtain next environment state $s_{t+1} \sim \mathcal{T}(s_t, a^{\text{refine}})$
         $t \leftarrow t + 1$
    end while
     $\{\tau'\} \leftarrow \{\tau'\} \cup \{\tau'\}$
     $\{\tau'_{\text{refine}}\} \leftarrow \{\tau'_{\text{refine}}\} \cup \{\tau'_{\text{refine}}\}$
until $G$ trajectories collected
return $\{\tau'\}, \{\tau'_{\text{refine}}\}$
Knowl 4 — Dyna-GRPO Online Reinforcement Learning Algorithm
algorithm
Dyna-GRPO strengthens an agent's simulation and decision-making capabilities without relying on external tree search during RL by iteratively alternating between simulation improvement (learning from future-state-guided rollouts) and direct policy improvement.
Input: Policy $\pi_\theta$ , environment $\mathcal{T}$ , group size $G$ , training iterations $N$ , simulation steps $n_T$ , policy steps $n_\pi$
Output: Updated policy $\pi_\theta$
for iteration $= 1$ to $N$ do
    // Simulation Improvement Phase
    for step $= 1$ to $n_T$ do
         $\{\tau'\}, \{\tau'_{\text{refine}}\} \leftarrow \text{SimRollout}(\pi_\theta, \mathcal{T}, G / 2)$
         $\{\tau\} \leftarrow \text{Rollout}(\pi_\theta, \mathcal{T}, G / 2)$
        Update $\pi_\theta$ with GRPO on combined unconditioned group $\{\tau\} \cup \{\tau'\}$
        Update $\pi_\theta$ with GRPO on future-conditioned group $\{\tau'_{\text{refine}}\}$ using advantage $A_{\text{refine}}$
    end for

    // Policy Improvement Phase
    for step $= 1$ to $n_\pi$ do
         $\{\tau\} \leftarrow \text{Rollout}(\pi_\theta, \mathcal{T}, G)$
        Update $\pi_\theta$ with standard GRPO on $\{\tau\}$
    end for
end for
return $\pi_\theta$
Knowl 5 — Advantage Formulations and Objectives in Dyna-GRPO
equation
Dyna-GRPO builds upon Group Relative Policy Optimization (GRPO). The standard GRPO surrogate objective is defined as:

$\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{\tau \sim \pi_{\theta_{\text{old}}}} \left[ \frac{1}{GT} \sum_{i=1}^G \sum_{t=1}^T \min \left( \rho_\theta(a_t^{(i)}) A(a_t^{(i)}), \text{clip}(\rho_\theta(a_t^{(i)}), 1 - \epsilon, 1 + \epsilon) A(a_t^{(i)}) \right) - \beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\theta_{\text{ref}}}) \right]$

where $\rho_\theta(a) = \frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{ref}}}(a|s)}$ is the importance sampling ratio, $G$ is the group size, $T$ is the trajectory horizon, $\epsilon$ is the clipping parameter, $\beta$ is the KL divergence penalty coefficient, and $\pi_{\theta_{\text{ref}}}$ is the reference policy.

The standard episode-level advantage function $A(a_t^{(i)}) = A_{\text{GRPO}}(\tau^{(i)})$ across a group of $G$ rollouts is normalized by group reward statistics:

$A_{\text{GRPO}}(\tau^{(i)}) = \frac{R(\tau^{(i)}) - \text{mean}(\{R(\tau^{(j)})\}_{j=1}^G)}{\text{std}(\{R(\tau^{(j)})\}_{j=1}^G)}, \quad R(\tau^{(i)}) = \sum_{t=1}^T R(s_t, a_t)$

where $R(s_t, a_t) = -0.1$ for non-terminal steps, and for terminal steps $R(s_T, a_T) = 10.0$ on success or $0.0$ on failure.

In the simulation improvement phase, the refined trajectory group $\{\tau^{(i)}_{\text{refine}}\}_{i=1}^{G/2}$ is trained using a specialized indicator advantage $A_{\text{refine}}$ :

$A_{\text{refine}}(\tau^{(i)}_{\text{refine}}) = \begin{cases} 1.0, & \text{if } \tau^{(i)}_{\text{refine}} \text{ is correct and } R(\tau^{(i)}_{\text{refine}}) > \max(\bar{R}, \bar{R}_{\text{refine}}) \\ 0.0, & \text{otherwise} \end{cases}$

where $\bar{R} = \frac{1}{G/2} \sum_{i=1}^{G/2} R(\tau^{(i)})$ is the empirical mean reward of standard unconditioned rollouts, and $\bar{R}_{\text{refine}} = \frac{1}{G/2} \sum_{i=1}^{G/2} R(\tau^{(i)}_{\text{refine}})$ is the mean reward of the SIMROLLOUT trajectories.
Knowl 6 — Simulation Score Protocol for Evaluating Agent World Modeling
definition
The Simulation Score (Sim Score $\in [0, 1]$ ) quantifies the fidelity and utility of an agent's internal world simulation during its reasoning process.

Given an agent's generated response $a_t \sim \pi_\theta(\cdot|s_t)$ at state $s_t$ :
1. An extraction LLM parses $a_t$ into an intended action plan $(\hat{a}_1, \dots, \hat{a}_d)$ and corresponding imagined next-state descriptions $(\hat{s}_{t+1}, \dots, \hat{s}_{t+d})$ .
2. The action plan is executed in the actual environment to observe true next-states $(s_{t+1}, \dots, s_{t+d})$ .
3. An independent judge LLM (e.g., Qwen3-235B-A22B-Instruct) evaluates the imagined states against ground-truth states across two sub-dimensions:
  - Correctness (maximum $0.3$ points): Evaluates factual agreement between imagined coordinates, objects, and environment layout relative to the reference observation.
  - Progress (maximum $0.7$ points): Evaluates whether the plan achieves meaningful task advancement (e.g., moving objects closer to targets, solving the task, or discovering required objects).
4. The sum of correctness and progress scores yields a turn score $\in [0, 1]$ , which is averaged over all steps in a trajectory to obtain the trajectory-level Simulation Score.

Knowl 7 — Task Success and Efficiency of Dyna-Mind on Sokoban and ALFWorld

data/table

Across both Sokoban (spatial planning) and ALFWorld (text-based embodied household tasks), RESIM and Dyna-Mind achieve high task success rates while maintaining significantly smaller token generation costs compared to standard reasoning model distillation (DISTILL(R1)). All Stage 1 and Stage 2 models use Qwen2.5-7B-Instruct as the base backbone.

Method	Gen. Token	Sokoban			ALFWorld
		ID	OOD	AVG	ID	OOD	AVG
REACT(Qwen2.5-7B-Instruct)	1.0x	$25.8_{\pm 1.8}$	-	-	$35.4_{\pm 1.9}$	-	-
REACT(Qwen2.5-32B-Instruct)	2.7x	$36.7_{\pm 4.2}$	-	-	$36.2_{\pm 3.3}$	-	-
REACT(GPT-4o)	1.5x	$37.8_{\pm 1.0}$	-	-	$51.3_{\pm 2.1}$	-	-
REACT(Claude-3.7-Sonnet)	2.3x	$70.3_{\pm 1.2}$	-	-	$46.1_{\pm 1.0}$	-	-
REACT(DeepSeek-V3)	2.5x	$57.0_{\pm 1.6}$	-	-	$55.2_{\pm 1.0}$	-	-
REACT(DeepSeek-R1)	14.5x	$96.6_{\pm 0.2}$	-	-	$62.5_{\pm 0.5}$	-	-
RESIM	2.0x	$96.4_{\pm 0.2}$	-	-	$87.7_{\pm 1.1}$	-	-
Dyna-Think
DIT(R1)+DDT( $\hat{\mathcal{T}}$ )	24.2x	$74.0_{\pm 1.4}$	$57.5_{\pm 1.2}$	$65.8_{\pm 1.9}$	$63.2_{\pm 1.5}$	$56.7_{\pm 2.8}$	$58.9_{\pm 2.3}$
Dyna-Mind Stage 1 (SFT)
DISTILL(V3)	2.1x	$49.2_{\pm 1.1}$	$34.4_{\pm 1.3}$	$41.8_{\pm 1.1}$	$58.9_{\pm 1.1}$	$56.7_{\pm 1.0}$	$57.8_{\pm 1.2}$
DISTILL(R1)	24.0x	$72.5_{\pm 2.9}$	$57.0_{\pm 1.9}$	$64.8_{\pm 2.5}$	$59.4_{\pm 1.5}$	$54.2_{\pm 3.9}$	$56.8_{\pm 3.5}$
DISTILL(RESIM)	2.0x	$71.9_{\pm 1.5}$	$55.5_{\pm 1.6}$	$63.7_{\pm 1.9}$	$78.9_{\pm 2.1}$	$69.3_{\pm 1.3}$	$74.1_{\pm 1.8}$
Dyna-Mind Stage 2 (RL)
DISTILL(RESIM) + RLOO	2.2x	$78.1_{\pm 1.8}$	$65.1_{\pm 1.3}$	$71.3_{\pm 0.9}$	$85.9_{\pm 1.3}$	$85.4_{\pm 2.0}$	$85.5_{\pm 2.0}$
DISTILL(RESIM) + GRPO	2.1x	$79.1_{\pm 1.3}$	$67.8_{\pm 0.6}$	$73.1_{\pm 1.4}$	$87.0_{\pm 3.2}$	$87.1_{\pm 1.1}$	$87.0_{\pm 1.8}$
DISTILL(RESIM) + DYNA-GRPO	1.9x	$82.5_{\pm 1.5}$	$70.1_{\pm 1.6}$	$77.1_{\pm 1.7}$	$92.5_{\pm 0.8}$	$89.1_{\pm 1.3}$	$90.8_{\pm 0.9}$

Key observations:

In ALFWorld, where DeepSeek-R1 struggles to simulate environment dynamics ( $62.5\%$ success), RESIM reaches $87.7\%$ , and DISTILL(RESIM) ( $74.1\%$ avg) outperforms DISTILL(R1) ( $56.8\%$ avg) while using $11\times$ fewer tokens.
Stage 2 online RL with DYNA-GRPO improves DISTILL(RESIM) to $77.1\%$ on Sokoban and $90.8\%$ on ALFWorld, exceeding standard GRPO and RLOO baselines.

Knowl 8 — Empirical Correlation Between Simulation Ability and Agent Task Success

data/table

Evaluating agent simulation fidelity reveals a strong positive Spearman correlation ( $r_s$ ) between Simulation Score and task success rate across both Sokoban and ALFWorld benchmarks.

Method	Sokoban		ALFWorld
	Success	Sim Score	Success	Sim Score
REACT(Qwen2.5-7B-Instruct)	$25.8_{\pm 1.8}$	$0.21$ ( $r=0.64$ )	$35.4_{\pm 1.9}$	$0.18$ ( $r=0.46$ )
REACT(DeepSeek-V3)	$57.0_{\pm 1.6}$	$0.54$ ( $r=0.81$ )	$55.2_{\pm 1.0}$	$0.35$ ( $r=0.68$ )
REACT(DeepSeek-R1)	$96.6_{\pm 0.2}$	$0.93$ ( $r=0.96$ )	$62.5_{\pm 0.5}$	$0.36$ ( $r=0.70$ )
RESIM	$96.4_{\pm 0.2}$	$1.00$ ( $-$ )	$87.7_{\pm 1.1}$	$1.00$ ( $-$ )
Dyna-Think
DIT(R1)+DDT( $\hat{\mathcal{T}}$ )	$74.0_{\pm 1.4}$	$0.62$ ( $r=0.74$ )	$63.2_{\pm 1.5}$	$0.36$ ( $r=0.76$ )
Dyna-Mind Stage 1 (SFT)
DISTILL(R1)	$72.5_{\pm 2.9}$	$0.61$ ( $r=0.75$ )	$59.4_{\pm 1.5}$	$0.34$ ( $r=0.77$ )
DISTILL(RESIM)	$71.9_{\pm 1.5}$	$0.62$ ( $r=0.78$ )	$78.9_{\pm 2.1}$	$0.37$ ( $r=0.74$ )
Dyna-Mind Stage 2 (RL)
DISTILL(RESIM) + GRPO	$79.1_{\pm 1.3}$	$0.62$ ( $r=0.65$ )	$87.0_{\pm 3.2}$	$0.38$ ( $\rho=0.48$ )
DISTILL(RESIM) + DYNA-GRPO	$82.5_{\pm 1.5}$	$0.67$ ( $r=0.64$ )	$92.5_{\pm 0.8}$	$0.43$ ( $r=0.55$ )

The measurements demonstrate:

DeepSeek-R1 achieves high simulation accuracy ( $0.93$ ) and task success ( $96.6\%$ ) in Sokoban, but drops to $0.36$ Sim Score and $62.5\%$ success in ALFWorld where layout modeling is challenging.
DYNA-GRPO increases both the Simulation Score (e.g., from $0.37$ to $0.43$ on ALFWorld) and success rate (from $78.9\%$ to $92.5\%$ on ALFWorld ID), showing that improving the agent's internal simulation capability directly boosts downstream task performance.

Knowl 9 — Performance of Dyna-Mind on the AndroidWorld Benchmark

data/table

Dyna-Mind evaluated on the realistic AndroidWorld GUI automation benchmark using screenshot-only observations and a maximum budget of 15 steps.

Method	Gen. Token	AndroidWorld
		ID	OOD	AVG
REACT(GPT-4o)	1.0x	$5.1_{\pm 0.2}$	-	-
REACT(Qwen2.5-VL-7B-Instruct)	1.0x	$5.3_{\pm 0.2}$	-	-
REACT(Qwen2.5-VL-72B-Instruct)	1.1x	$19.5_{\pm 0.4}$	-	-
RESIM	2.1x	$34.4_{\pm 0.4}$	-	-
Dyna-Mind Stage 1 (SFT)
DISTILL-7B(Qwen2.5-VL-72B-Instruct)	1.0x	$13.1_{\pm 0.4}$	$8.6_{\pm 0.2}$	$10.8_{\pm 0.6}$
DISTILL-7B(RESIM)	2.1x	$21.1_{\pm 0.4}$	$10.2_{\pm 0.6}$	$15.7_{\pm 0.8}$
DISTILL-32B(RESIM)	2.0x	$32.8_{\pm 0.4}$	$15.6_{\pm 0.7}$	$24.2_{\pm 0.6}$
Dyna-Mind Stage 2 (RL)
DISTILL-32B(RESIM) + GRPO	2.1x	$35.3_{\pm 0.4}$	$20.3_{\pm 0.6}$	$27.8_{\pm 0.4}$
DISTILL-32B(RESIM) + DYNA-GRPO	1.9x	$40.7_{\pm 1.0}$	$22.9_{\pm 1.0}$	$31.8_{\pm 1.0}$

Key takeaways:

RESIM inference achieves $34.4\%$ on ID tasks, substantially exceeding direct prompting of Qwen2.5-VL-72B ( $19.5\%$ ) and GPT-4o ( $5.1\%$ ).
Distilling RESIM into Qwen2.5-VL-32B achieves $24.2\%$ average success across ID and OOD splits.
Stage 2 online RL with DYNA-GRPO achieves $31.8\%$ overall average ( $40.7\%$ ID, $22.9\%$ OOD), outperforming standard GRPO ( $27.8\%$ ).

Knowl 10 — Effect of Ground-Truth Next-State Feedback on Decision Making

data/table

Providing agents with intermediate ground-truth next-state observations during inference (via SIMROLLOUT) significantly boosts decision-making performance over standard REACT prompting across both open and proprietary base models.

Base Model	Method	Sokoban	ALFWorld
Qwen2.5-7B-Instruct	REACT	$25.8_{\pm 1.8}$	$35.4_{\pm 1.9}$
	SIMROLLOUT	$30.0_{\pm 1.4}$	$39.1_{\pm 1.6}$
GPT-4o-2024-11-20	REACT	$37.8_{\pm 1.0}$	$51.3_{\pm 2.1}$
	SIMROLLOUT	$41.4_{\pm 1.2}$	$64.8_{\pm 2.5}$
GPT-4.1	REACT	$67.9_{\pm 1.0}$	$54.4_{\pm 2.1}$
	SIMROLLOUT	$71.1_{\pm 1.3}$	$67.9_{\pm 2.0}$

These results confirm that grounding the agent's thinking process in real next-state feedback enables error correction and reduces execution failures, providing the empirical foundation for using SIMROLLOUT to generate training data during online RL.

Coverage note — None was omitted; all primary methods, algorithms (ReSim, SimRollout, Dyna-GRPO), theoretical/equation formulations, evaluation definitions, and main empirical tables from both text and multimodal benchmarks are covered.

References

1.Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like a human, 2024. URL https://arxiv.org/abs/2410.08164.
2.Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s2: A compositional generalist-specialist framework for computer use agents, 2025. URL https://arxiv.org/abs/2504.00906.
3.Anthropic. Claude 3.7 Sonnet and Claude Code. https://www.anthropic.com/news/claude-3-7-sonnet, 2025. Accessed: 2025-05-13.
4.Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https://arxiv.org/abs/2502.13923.
5.M.S. Bennett. A Brief History of Intelligence: Evolution, AI, and the Five Breakthroughs That Made Our Brains. HarperCollins, 2023. ISBN 9780063286368. URL https://books.google.com/books?id=tymCEAAAQBAJ.
6.Hyungjoo Chae, Namyoung Kim, Kai Tzu iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo. Web agents with world models: Learning and leveraging environment dynamics in web navigation, 2025. URL https://arxiv.org/abs/2410.13232.
7.Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-flan: Designing data and methods of effective agent tuning for large language models, 2024. URL https://arxiv.org/abs/2403.12881.
8.Nathaniel D Daw and Peter Dayan. The algorithmic anatomy of model-based evaluation. Philosophical Transactions of the Royal Society B: Biological Sciences, 369(1655):20130478, 2014.
9.Nathaniel D. Daw, Yael Niv, and Peter Dayan. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8:1704–1711, 2005. URL https://api.semanticscholar.org/CorpusID:16385268.
10.DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, and et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025a. URL https://arxiv.org/abs/2501.12948.
11.DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, and et al. Deepseek-v3 technical report, 2025b. URL https://arxiv.org/abs/2412.19437.
12.Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023. URL https://arxiv.org/abs/2306.06070.
13.Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents, 2025. URL https://arxiv.org/abs/2506.11763.
14.Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent training, 2025. URL https://arxiv.org/abs/2505.10978.
15.Boyu Gou, Zanming Huang, Yuting Ning, Yu Gu, Michael Lin, Weijian Qi, Andrei Kopanev, Botao Yu, Bernal Jiménez Gutiérrez, Yiheng Shu, Chan Hee Song, Jiaman Wu, Shijie Chen, Hanane Nour Moussa, Tianshu Zhang, Jian Xie, Yifei Li, Tianci Xue, Zeyi Liao, Kai Zhang, Boyuan Zheng, Zhaowei Cai, Viktor Rozgic, Morteza Ziyadi, Huan Sun, and Yu Su. Mind2web 2: Evaluating agentic search with agent-as-a-judge, 2025a. URL https://arxiv.org/abs/2506.21506.
16.Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents, 2025b. URL https://arxiv.org/abs/2410.05243.
17.Yu Gu, Kai Zhang, Yuting Ning, Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, Huan Sun, and Yu Su. Is your llm secretly a world model of the internet? model-based planning for web agents, 2025. URL https://arxiv.org/abs/2411.06559.
18.Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023. URL https://arxiv.org/abs/2312.06674.
19.Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL https://arxiv.org/abs/2310.06770.
20.Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents, 2024. URL https://arxiv.org/abs/2407.01476.
21.Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 REINFORCE samples, get a baseline for free!, 2019. URL https://openreview.net/forum?id=r1lgTGL5DE.
22.Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task, 2024. URL https://arxiv.org/abs/2210.13382.
23.Haowei Liu, Xi Zhang, Haiyang Xu, Yuyang Wanyan, Junyang Wang, Ming Yan, Ji Zhang, Chunfeng Yuan, Changsheng Xu, Weiming Hu, and Fei Huang. Pc-agent: A hierarchical multi-agent collaboration framework for complex task automation on pc, 2025. URL https://arxiv.org/abs/2502.14282.
24.Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models, 2023. URL https://arxiv.org/abs/2309.00941.
25.OpenAI. New and improved content moderation tooling. https://openai.com/index/new-and-improved-content-moderation-tooling/, 2022. Accessed: 2025-05-13.
26.OpenAI. Hello GPT-4o. https://openai.com/index/hello-gpt-4o/, 2024. Accessed: 2024-09-28.
27.OpenAI. Introducing GPT-4.1 in the api. https://openai.com/index/gpt-4-1/, 2025. Accessed: 2025-09-17.
28.Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, Kam-Fai Wong, and Shang-Yu Su. Deep dyna-q: Integrating planning for task-completion dialogue policy learning, 2018. URL https://arxiv.org/abs/1801.06176.
29.Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. Ui-tars: Pioneering automated gui interaction with native agents, 2025. URL https://arxiv.org/abs/2501.12326.
30.Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115.
31.Qwen Team. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.
32.Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. Androidworld: A dynamic benchmarking environment for autonomous agents, 2025. URL https://arxiv.org/abs/2405.14573.
33.Max-Philipp B. Schrader. gym-sokoban. https://github.com/mpSchrader/gym-sokoban, 2018.
34.Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588 (7839):604–609, December 2020. ISSN 1476-4687. doi: 10.1038/s41586-020-03051-4. URL http://dx.doi.org/10.1038/s41586-020-03051-4.
35.Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300.
36.Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. URL https://arxiv.org/abs/2303.11366.
37.Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity, 2025. URL https://arxiv.org/abs/2506.06941.
38.Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning, 2021. URL https://arxiv.org/abs/2010.03768.
39.Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bull., 2(4):160–163, July 1991. ISSN 0163-5719. doi: 10.1145/122344.122377. URL https://doi.org/10.1145/122344.122377.
40.Edward C Tolman. Cognitive maps in rats and men. Psychological review, 55(4):189, 1948.
41.Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models, 2023. URL https://arxiv.org/abs/2305.16291.
42.Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, Zhennan Shen, Zhuokai Li, Ryan Li, Xiaochuan Li, Junda Chen, Boyuan Zheng, Peihang Li, Fangyu Lei, Ruisheng Cao, Yeqiao Fu, Dongchan Shin, Martin Shin, Jiarui Hu, Yuyan Wang, Jixuan Chen, Yuxiao Ye, Danyang Zhang, Dikang Du, Hao Hu, Huarong Chen, Zaida Zhou, Haotian Yao, Ziwei Chen, Qizheng Gu, Yipu Wang, Heng Wang, Diyi Yang, Victor Zhong, Flood Sung, Y. Charles, Zhilin Yang, and Tao Yu. Opencua: Open foundations for computer-use agents, 2025a. URL https://arxiv.org/abs/2508.09123.
43.Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning, 2025b. URL https://arxiv.org/abs/2504.20073.
44.Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution, 2025a. URL https://arxiv.org/abs/2502.18449.
45.Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, Hyokun Yun, and Lihong Li. Webagent-r1: Training web agents via end-to-end multi-turn reinforcement learning, 2025b. URL https://arxiv.org/abs/2505.16421.
46.Yuexin Wu, Xiujun Li, Jingjing Liu, Jianfeng Gao, and Yiming Yang. Switch-based active deep dyna-q: Efficient adaptive planning for task-completion dialogue policy learning, 2018. URL https://arxiv.org/abs/1811.07550.
47.Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024. URL https://arxiv.org/abs/2404.07972.
48.Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction, 2025. URL https://arxiv.org/abs/2412.04454.
49.John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering, 2024. URL https://arxiv.org/abs/2405.15793.
50.Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023a. URL https://arxiv.org/abs/2305.10601.
51.Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023b. URL https://arxiv.org/abs/2210.03629.
52.Xiao Yu, Maximillian Chen, and Zhou Yu. Prompt-based monte-carlo tree search for goal-oriented dialogue policy planning, 2023. URL https://arxiv.org/abs/2305.13660.
53.Xiao Yu, Baolin Peng, Vineeth Vajipey, Hao Cheng, Michel Galley, Jianfeng Gao, and Zhou Yu. Exact: Teaching ai agents to explore with reflective-mcts and exploratory learning, 2025a. URL https://arxiv.org/abs/2410.02052.
54.Xiao Yu, Baolin Peng, Ruize Xu, Michel Galley, Hao Cheng, Suman Nath, Jianfeng Gao, and Zhou Yu. Dyna-think: Synergizing reasoning, acting, and world model simulation in ai agents, 2025b. URL https://arxiv.org/abs/2506.00320.
55.Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms, 2023. URL https://arxiv.org/abs/2310.12823.
56.Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, Zhiwei Liu, Yihao Feng, Tulika Awalgaonkar, Rithesh Murthy, Eric Hu, Zeyuan Chen, Ran Xu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, and Caiming Xiong. xlam: A family of large action models to empower ai agent systems, 2024. URL https://arxiv.org/abs/2409.03215.
57.Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is a generalist web agent, if grounded, 2024. URL https://arxiv.org/abs/2401.01614.
58.Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https://arxiv.org/abs/2306.05685.
59.Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models, 2024a. URL https://arxiv.org/abs/2310.04406.
60.Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024b. URL https://arxiv.org/abs/2307.13854.
61.Lixin Zou, Long Xia, Pan Du, Zhuo Zhang, Ting Bai, Weidong Liu, Jian-Yun Nie, and Dawei Yin. Pseudo dyna-q: A reinforcement learning framework for interactive recommendation. In Proceedings of the 13th International Conference on Web Search and Data Mining, WSDM ’20, pp. 816–824, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450368223. doi: 10.1145/3336191.3371801. URL https://doi.org/10.1145/3336191.3371801.

Citation

MLA

Yu, X., et al. “Dyna-Mind: Learning to Simulate from Experience for Better AI Agents”. arXiv, 2025, http://arxiv.org/abs/2510.09577v1.

APA

Yu, X., Peng, B., Galley, M., Cheng, H., Wu, Q., Kulkarni, J., Nath, S., Yu, Z., & Gao, J. (2025). Dyna-Mind: Learning to Simulate from Experience for Better AI Agents. arXiv. http://arxiv.org/abs/2510.09577v1

Chicago

Yu, X., B. Peng, M. Galley, et al. 2025. “Dyna-Mind: Learning to Simulate from Experience for Better AI Agents”. arXiv. http://arxiv.org/abs/2510.09577v1.

Harvard

Yu, X. et al. (2025) “Dyna-Mind: Learning to Simulate from Experience for Better AI Agents”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2510.09577v1.

Vancouver

1. Yu X, Peng B, Galley M, Cheng H, Wu Q, Kulkarni J, Nath S, Yu Z, Gao J (2025) Dyna-Mind: Learning to Simulate from Experience for Better AI Agents. arXiv

BibTeX

@article{yu2025dyna,
  title = {Dyna-Mind: Learning to Simulate from Experience for Better AI Agents},
  author = {Yu, Xiao and Peng, Baolin and Galley, Michel and Cheng, Hao and Wu, Qianhui and Kulkarni, Janardhan and Nath, Suman and Yu, Zhou and Gao, Jianfeng},
  year = {2025},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2510.09577v1},
  eprint = {2510.09577}
}

Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Dyna-Mind: Learning to Simulate from Experience for Better AI Agents

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Dyna-Mind Framework for Grounded Simulation in Language Model Agents

Knowl 2 — Reasoning with Simulations (RESIM) Search Tree Aggregation and Distillation

Knowl 3 — Simulation Refinement Rollout (SIMROLLOUT)

Knowl 4 — Dyna-GRPO Online Reinforcement Learning Algorithm

Knowl 5 — Advantage Formulations and Objectives in Dyna-GRPO

Knowl 6 — Simulation Score Protocol for Evaluating Agent World Modeling

Knowl 7 — Task Success and Efficiency of Dyna-Mind on Sokoban and ALFWorld

Knowl 8 — Empirical Correlation Between Simulation Ability and Agent Task Success

Knowl 9 — Performance of Dyna-Mind on the AndroidWorld Benchmark

Knowl 10 — Effect of Ground-Truth Next-State Feedback on Decision Making

References

Citation

BibTeX

Access the Paper