TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents

Leitian Tao$^{1,2}$ Baolin Peng$^{2}$ Wenlin Yao$^{2}$ Tao Ge$^{2}$ Hao Cheng$^{2}$ Mike Hang Wang$^{2}$ Jianfeng Gao$^{2}$ Sharon Li$^{1}$
$^{1}$University of Wisconsin–Madison $^{2}$Microsoft Research

Abstract

Multi-turn agents solve complex tasks through extended sequences of tool interactions before producing a final answer, making credit assignment a fundamental challenge during post-training. Outcome rewards provide reliable supervision for short-horizon reasoning, but become sparse and high-variance as trajectories grow to tens or hundreds of tool calls. They can also be misleading: a failed rollout may contain many useful actions that move the agent closer to the goal, yet outcome-only training assigns them the same negative advantage as the eventual mistake. We propose TRACE (Turn-level Reward Assignment via Credit Estimation), a dense credit-assignment method for agentic reinforcement learning. TRACE represents rollouts as state transitions at tool-call boundaries, obtains gold-answer log-probabilities from a frozen reference model, transforms them into log-ratio state values, and derives per-action rewards as Temporal-Difference changes in those values. This requires no additional critic or process-label training, and its one-step log-ratio TD component telescopes across redundant tool calls. On long-horizon complex search, TRACE substantially improves base-model tool-use ability using pure RL, without a cold-start supervised fine-tuning stage, an agentic mid-training stage, or training on live-web data. On the closed-web BrowseComp-Plus benchmark, it raises Qwen3-4B from 7.2 to 35.6 and Qwen3-30B-A3B from 8.4 to 42.6. The learned search behavior also transfers to open-web benchmarks, and the learning curves show earlier improvement and faster convergence during RL training.

Executive Summary: TRACE addresses a core challenge in training AI agents that solve complex tasks through dozens of tool calls, such as web searches and document navigation. Outcome rewards based solely on the final answer become too sparse and noisy over long trajectories, often failing to distinguish useful intermediate actions from redundant or harmful ones and leading to slow or unstable learning.

The work set out to test whether turn-level credit signals derived from a frozen reference model could deliver dense, reliable rewards for agentic reinforcement learning without step labels, trained judges, critics, or process supervision.

The method represents each rollout as state transitions at tool-call boundaries. It scores how much more predictable the gold answer becomes after each interaction using log-probabilities from an unchanged reference model, converts these into log-ratio values that measure relative progress, and assigns rewards as temporal-difference changes between adjacent states. These dense signals are combined with standard outcome advantages and optimized jointly via group-relative policy optimization on synthetic multi-document search tasks.

TRACE raised Qwen3-4B performance on the closed-web BrowseComp-Plus benchmark from 7.2 to 35.6 and Qwen3-30B-A3B from 8.4 to 42.6. The same agents also improved on open-web benchmarks (reaching 12.9 on BrowseComp, 52.0 on GAIA, and 45.0 on xbench-DeepSearch for the 30B model). Learning curves showed earlier gains and faster convergence than outcome-only baselines, while controlled ablations confirmed that the log-ratio formulation and moderate turn-level weighting were responsible for the gains.

These results indicate that verifier-anchored turn credit can organize existing model capabilities into effective long-horizon search behavior using only final-answer correctness and pure RL, without cold-start supervised fine-tuning or live-web data. The approach therefore lowers the cost and complexity of developing capable agents while improving training stability.

Organizations developing agent systems should incorporate similar dense, reference-model credit signals into RL pipelines for multi-turn tool use. Priority should be given to tasks with compact, verifiable answers; for open-ended or long-form outputs, alternative progress estimators will be needed first. Additional experiments on diverse agent domains would strengthen the case for broad deployment.

The main limitations are that validation focused on search with short verifiable answers and a closed training corpus. Performance on open-ended tasks or live environments may vary, and the method’s advantage is clearest when trajectories are long enough for delayed credit to matter. Overall confidence in the reported gains for the tested setting is high given the controlled comparisons, but readers should treat broader claims as directional until further evidence appears.

1. Introduction

Section Summary: Large language model agents tackle complex, multi-step tasks such as web searches by making many tool calls, yet training them with rewards given only at the end fails to identify which intermediate actions actually helped reach the answer. TRACE solves this by using a fixed reference model to check how much each tool call increases the chance of predicting the correct final answer, then converting those changes into small positive or negative rewards at each step. The result is faster learning and substantially higher success on difficult search benchmarks, all without extra labels, judging models, or process supervision.

Large language model (LLM) agents increasingly solve complex tasks by reasoning and acting through many interactions with external environments, including web navigation, software engineering, and general computer use ([1, 2, 3, 4, 5, 6, 7, 8]). Reinforcement learning with verifiable rewards (RLVR) has been effective for single-turn reasoning tasks such as mathematics and coding, where a deterministic checker can evaluate the final answer and provide a clean outcome reward ([9, 10]). Applying the same idea to agents is appealing but insufficient: an agentic rollout may contain dozens of search, open, find, code, or other tool-use decisions before the final answer is verified. A single terminal reward says whether the trajectory succeeded, but it does not reveal which intermediate actions gathered necessary evidence, which were redundant, or which derailed the agent.

Dense turn-level rewards are therefore desirable, but obtaining them is challenging because partial progress in agent trajectories is not directly observed. As illustrated in Figure 1, a failed rollout can begin with useful search and open actions that make the answer easier to infer, while a successful rollout can contain redundant searches or accidental openings that do not contribute to the final answer. Outcome-only training assigns the same trajectory-level advantage to all of these turns, under-crediting productive exploration, over-crediting irrelevant actions, and increasing gradient variance as horizons grow ([11, 12, 13]). Prior work on process supervision offers finer feedback, but typically requires step-level labels, a strong LLM judge to score intermediate behavior ([14]), or a trained process reward model whose scores may drift away from final-answer correctness ([15, 16, 17, 18, 19]). This raises the central question of this work: how can we identify the tool calls that actually move a long-horizon agent toward the answer, and turn that signal into dense rewards without relying on step labels, a strong judge, or a trained process reward model?

We propose $\textsc{TRACE}$ (Turn-level Reward Assignment via Credit Estimation), a critic-free credit-assignment framework that keeps the final verifier as the anchor while adding dense credit at tool-call boundaries. The key idea is to use a frozen reference model not as a judge, but as a stable probe of whether each trajectory prefix makes the gold answer more predictable. $\textsc{TRACE}$ represents a rollout as state transitions at tool-call boundaries, scores each prefix by the reference model's gold-answer log-probability, and converts the score into a log-ratio state value measuring progress toward the answer. It then assigns turn rewards using temporal-difference (TD) changes between adjacent values: a tool call receives positive credit when its returned observation increases answer predictability, near-zero credit when it adds no useful evidence, and negative credit when it moves the trajectory away from the answer. Because the one-step TD credits telescope, redundant intermediate turns cannot inflate this component, while the cumulative credit remains aligned with the final reference-model state. The resulting rewards combine this dense TD signal with the standard outcome-level advantage ([20]), preserving verifiable success as the final training target while distinguishing which turns deserve credit.

$**Figure 1:** **Credit assignment at tool-call boundaries in a search trajectory.** The figure illustrates a long-horizon search rollout decomposed into tool-call turns. Early search and open actions can add task-relevant evidence to the transcript even when a later branch leads to an incorrect final answer. Outcome-reward training attaches one trajectory-level advantage to all actions in the rollout, whereas $\textsc{TRACE}$ computes prefix values at tool boundaries and assigns turn credit from adjacent value changes.$

We evaluate $\textsc{TRACE}$ on long-horizon complex search, a setting where agents must learn to interact with an environment, explore through multiple tool calls, and refine later actions from earlier observations. Because common multi-hop benchmarks can be solved by strong agents in only a few turns ([21]), we train on deeper synthetic search questions and evaluate on both closed-web and open-web deep-research benchmarks. Our recipe is pure RL: it uses outcome-level and turn-level rewards directly, without a cold-start supervised fine-tuning stage, an agentic mid-training stage, live-web training data, a strong judging model, or a trained process reward model. Despite this minimal supervision, $\textsc{TRACE}$ substantially improves base-model tool use in both closed-web and open-web settings. On closed-web BrowseComp-Plus ([22]), it raises Qwen3-4B from $7.2$ to $35.6$ and Qwen3-30B-A3B from $8.4$ to $42.6$. The learned behavior also transfers to open-web retrieval, with the 30B-A3B agent reaching $12.9$ on BrowseComp, $52.0$ on GAIA, and $45.0$ on xbench-DeepSearch. In addition to final performance, the training curves show that $\textsc{TRACE}$ starts improving earlier and converges faster, indicating that turn-level credit makes long-horizon tool use easier to learn from pure RL. These results show that verifier-anchored turn-level credit can teach base models to explore, interact with the environment, and refine tool use for complex long-horizon tasks, while remaining independent of expensive process supervision or strong judge-based feedback.

2. Preliminaries

Section Summary: In agentic reinforcement learning, an LLM policy solves tasks by making multiple tool calls that interleave with environment feedback before producing a final answer, and it is trained with an outcome reward that arrives only at the end of the entire sequence. This setup creates a difficult credit-assignment problem because the single terminal reward must supervise a long chain of decisions without indicating which individual tool uses helped or hurt the result, even when methods like group-relative policy optimization compare whole trajectories. Temporal-difference learning offers a way to measure incremental progress by comparing the estimated value of states before and after each action, but applying it here requires a reliable way to judge partial tool trajectories without training an extra critic or using step-by-step labels.

2.1 Agentic Reinforcement Learning

In agentic reinforcement learning, a policy LLM $\pi_\theta$ solves a prompt $x\sim\mathcal{D}$ by interleaving assistant tokens, tool calls from an available tool set $\mathcal{T}$, and tool observations before producing a final answer. A common KL-regularized training objective is

$ \max_{\pi_\theta} \mathbb{E}{x \sim \mathcal{D}, , \tau \sim \pi\theta(\cdot \mid x;\mathcal{T})} \left[r_\phi(x, \tau) \right]

\beta_{\mathrm{KL}}, \mathbb{D}{\mathrm{KL}}!\left[\pi\theta(\tau \mid x;\mathcal{T}) , |, \pi_{\mathrm{ref}}(\tau \mid x;\mathcal{T}) \right],\tag{1} $

where $\tau$ denotes the complete rollout, $r_\phi$ is the reward function, $\pi_{\mathrm{ref}}$ is a reference policy, and $\beta_{\mathrm{KL}}$ controls the strength of the KL penalty. In reinforcement learning with verifiable rewards, $r_\phi$ is usually an outcome reward that is observed only after the rollout is complete, for example by checking whether the final answer satisfies an automatically verifiable criterion ([11, 9]). Unlike single-turn reasoning, agentic rollouts induce a distribution over both the intermediate interaction trajectory and the final answer ([1, 2, 3, 4]). We write the interaction part as alternating policy actions and environment observations. Let $\mathcal{R}=((a_1, o_1), \ldots, (a_{T_{\mathcal{R}}}, o_{T_{\mathcal{R}}}))$ and $H_k=(x, a_1, o_1, \ldots, a_{k-1}, o_{k-1})$. Then

$ P_\theta(\mathcal{R}, y \mid x;\mathcal{T}) = \underbrace{ \prod_{k=1}^{T_{\mathcal{R}}} \pi_\theta(a_k \mid H_k;\mathcal{T}), P_{\mathrm{env}}(o_k \mid H_k, a_k;\mathcal{T}) }{\mathrm{Tool\ interaction}} \cdot \underbrace{ \prod{t=1}^{T_y} \pi_\theta(y_t \mid y_{<t}, H_{T_{\mathcal{R}}+1};\mathcal{T}) }_{\mathrm{Answer\ generation}},\tag{2} $

where $\mathcal{R}$ is the multi-turn trajectory, including tool-call feedback, and $y$ is the final answer. Gradients are taken through assistant actions and answer tokens, while tool observations are environment outputs rather than policy-generated tokens. This decomposition highlights the source of the credit-assignment problem: the terminal reward supervises a long sequence of heterogeneous policy decisions, but does not identify which turns made the answer more or less likely.

For credit assignment, it is useful to view the interaction trajectory at tool-call boundaries. Given a rollout with $T_{\mathcal{R}}$ tool interactions, let $S_k$ denote the prefix containing the prompt and the first $k$ action–observation pairs, optionally followed by the final-answer opener used to score answer tokens. The transition from $S_k$ to $S_{k+1}$ is induced by the next policy action and environment observation $(a_{k+1}, o_{k+1})$. These prefix transitions are the natural units for assigning credit to tool use, because they isolate how each interaction changes the information available to the agent.

Typical RL training builds on outcome-based agentic RL algorithms such as Group Relative Policy Optimization (GRPO) ([10]), which compare multiple rollouts from the same prompt and avoid a learned critic for the outcome-reward stage ([9]). However, the resulting group-relative advantage is still attached to the entire rollout. It can rank a successful trajectory above failed alternatives, but it cannot distinguish useful tool calls from redundant or harmful ones within the same trajectory. This limitation motivates a reward construction that preserves the reliability of outcome verification while exposing denser signal at the prefix-transition level.

2.2 Temporal-Difference Credit Assignment

Temporal-difference (TD) learning estimates progress by comparing the value of two adjacent states ([20]). Given a value function $V(s)$, an immediate reward $r_t$, and a discount $\gamma$, the one-step TD error is

$ \delta_t ;=; r_t + \gamma V(s_{t+1}) - V(s_t).\tag{3} $

The TD error is positive when the transition leads to a state whose realized or estimated future return is higher than expected, and negative when it leads to a worse state. In sparse-reward agentic tasks, intermediate environment rewards are usually unavailable. If we set $r_t=0$ before termination and use $\gamma = 1$, the TD form reduces to the value difference $V(s_{t+1}) - V(s_t)$, which naturally measures the incremental progress made by one transition. The difficulty is that long-horizon agent tasks usually lack a reliable dense value function over partially completed tool trajectories. The next section addresses this missing ingredient: how to obtain a stable prefix-level progress signal without training a critic or collecting step-level process labels.

3. Turn-Level Reward Assignment via Credit Estimation

Section Summary: TRACE assigns credit in long agent rollouts by splitting trajectories at tool-call boundaries and scoring each prefix with a frozen reference model that measures how much easier the gold answer has become to predict. These scores are turned into log-ratio state values, and the one-step changes between consecutive prefixes become turn-level rewards that reflect whether a given tool interaction helped or hurt progress toward the answer. The resulting turn credits are simply added to the standard final-outcome advantage so the policy is updated with both fine-grained process signals and the overall task result.

$**Figure 2:** **$\textsc{TRACE}$ reward construction at tool-call boundaries.** The figure shows a long-horizon rollout segmented by tool calls and observations. Each prefix state is scored by a frozen reference model, transformed into a log-ratio value, and compared with adjacent prefix values to obtain turn-level TD credit. The turn-level advantage is then combined with the trajectory-level outcome advantage for policy optimization.$

We introduce $\textsc{TRACE}$, a critic-free credit-assignment method for long-horizon agent post-training. Instead of learning a critic over sparse terminal rewards, $\textsc{TRACE}$ measures whether each tool interaction makes the gold answer more predictable under a frozen reference model. It splits rollouts at tool-call boundaries, maps prefix answer log-probabilities to log-ratio state values, uses TD changes in those values as turn-level rewards, and mixes the resulting turn-level advantage with the standard GRPO outcome advantage. The outcome reward remains the final arbiter of task success. Figure 2 illustrates the framework, and Algorithm 1 summarizes the procedure.

3.1 Tool-Boundary States

Following the prefix-transition view in Section 2.1, we instantiate the notation used by $\textsc{TRACE}$. For a sampled trajectory on question $x$ with gold answer $y^\star$, write

$ \tau =\bigl(x, ;a_1, ;o_1, ;a_2, ;o_2, ;\ldots, ; a_T, ;o_T, ;\hat{y}\bigr),\tag{4} $

where $a_k$ is the $k$-th assistant action, $o_k$ is the corresponding tool observation, $\hat{y}$ is the final answer span, and $T$ is the number of tool interactions. The terminal verifier returns $R=R(\hat{y}, y^\star)$. $\textsc{TRACE}$ constructs one credit target for each transition $S_k\to S_{k+1}$ induced by $(a_{k+1}, o_{k+1})$, while the policy-gradient loss is applied only to policy-generated assistant actions and answer tokens; tool observations remain environment outputs and are masked from the loss. The remaining question is how to score whether a prefix transition improves the trajectory's ability to reach the gold answer.

3.2 State Value via Log-Ratio Answer Probability

Reference answer score.

Long-horizon agentic RL makes critic learning brittle: the trajectory reward is sparse, observations shift the state after each tool action, and failed trajectories can still contain useful action–observation transitions. $\textsc{TRACE}$ therefore replaces a learned critic with a frozen reference model $\pi_{\mathrm{ref}}$ (a copy of the policy initialization), used only to estimate how ready the single-trajectory prefix is to produce the gold answer. This is a natural proxy for progress in RLVR settings: the gold answer is available during training, and a transcript that has collected relevant evidence should make that answer easier for a fixed language model to conditionally predict. Freezing the model keeps this probe stable across policy updates and prevents it from becoming another learned reward model. For state $S_k$, the average gold-answer log-probability is

$ \bar{\ell}k =\frac{1}{|y^\star|}\sum{t=1}^{|y^\star|} \log \pi_{\mathrm{ref}}!\left(y^\star_t \mid S_k, , y^\star_{<t}\right) ;\le; 0 .\tag{5} $

Larger (less negative) $\bar{\ell}_k$ means that the current transcript already contains more evidence for generating $y^\star$. All $\bar{\ell}_k$ are obtained in a single batched forward pass and are never optimized; final-answer correctness is still determined by the outcome verifier.

Log-ratio state value.

The score $\bar{\ell}_k$ is a prefix-readiness measure, but its raw scale is not directly suitable for credit assignment. Raw changes in $\bar{\ell}_k$ can be misleading: the same absolute gain is minor when the trajectory is far from the answer but important when it nearly resolves the answer. For instance, shrinking the remaining gap from $0.2$ to $0.1$ removes half the uncertainty, whereas $5.1!\to!5.0$ barely changes it despite an identical raw gain. We therefore model the state value as relative closure of the initial answer-likelihood gap. Define $d_k=-\bar{\ell}_k+\epsilon$ with offset $\epsilon>0$, and set

$ V(S_k);=;\log\frac{d_0}{d_k} ;=;\log\frac{-\bar{\ell}_0+\epsilon}{-\bar{\ell}_k+\epsilon}.\tag{6} $

Thus $V(S_0)=0$, and a larger $V(S_k)$ means that the action–observation history has closed a larger fraction of the initial gap. The later TD difference $V(S_{k+1})-V(S_k)$ therefore measures the value added by the transition induced by action $a_{k+1}$ and observation $o_{k+1}$.

$**Algorithm 1:** $\textsc{TRACE}$ reward construction and policy update for one prompt group.$

3.3 Turn-Level Reward via Temporal-Difference Credit

One-step TD action credit.

With no intermediate environment reward and discount $1$, the credit assigned to the transition $S_k\xrightarrow{a_{k+1}, , o_{k+1}}S_{k+1}$ is the one-step TD change in value,

$ \delta_k ;=; V(S_{k+1})-V(S_k) ;=;\log\frac{d_k}{d_{k+1}}, \qquad k=0, \dots, T-1 .\tag{7} $

This is positive when the action and returned observation make the gold answer more likely, equals $\log 2$ when the transition halves the remaining gap, is $0$ when the gap is unchanged, and is negative when the transition moves away from the answer. Because the credits telescope,

$ \sum_{k=0}^{T-1}\delta_k ;=; V(S_T)-V(S_0) ;=;\log\frac{-\bar{\ell}_0+\epsilon}{-\bar{\ell}_T+\epsilon}, $

so total credit depends only on the endpoints: redundant intermediate transitions cannot inflate it, and the agent is not rewarded for padding a trajectory. Appendix A.2 compares this log-ratio credit with raw and linearized alternatives.

$K$-step turn credit.

The one-step TD credit above can miss delayed tool effects: for example, a browser.search may only surface candidate links, while the answer likelihood jumps after a later browser.open exposes decisive evidence. We therefore use a truncated $K$-step TD backup with $K\ge 1$: each turn receives discounted future TD progress from the current transition and the next few transitions in the same trajectory. For rollout $g$ and turn $k$, let

$ h_{g, k}=\min(k+K-1, , T_g-1)\tag{8} $

be the last transition included in the backup window. The local progress credit assigned to the current turn is

$ c^{(K)}{g, k} =\frac{1}{Z{g, k}} \sum_{u=k}^{h_{g, k}}\gamma_{\mathrm{td}}^{, u-k}\delta_{g, u}, \qquad Z_{g, k}=\sum_{u=k}^{h_{g, k}}\gamma_{\mathrm{td}}^{, u-k},\tag{9} $

where $\gamma_{\mathrm{td}}$ discounts delayed evidence and $Z_{g, k}$ is the $K$-step normalization factor used in our reported runs. In ablations, we use $K=0$ only as a shorthand for disabling the dense TD backup entirely, not as an input to 9. When the backup window reaches the end of the trajectory, we additionally anchor the last turns to the verifiable outcome signal:

$ r^{\mathrm{turn}}{g, k} =c^{(K)}{g, k} +\mathbf{1}!\left[h_{g, k}=T_g-1\right] \lambda_{\mathrm{term}}, \gamma_{\mathrm{td}}^{, T_g-k}, A^{\mathrm{out}}_g .\tag{10} $

Here $A^{\mathrm{out}}g$ is the standard GRPO group-relative advantage computed from the terminal reward, and $\lambda{\mathrm{term}}$ controls the strength of the terminal anchor. The exponent $T_g-k$ treats final-answer generation as one additional transition after the last tool observation, so the last tool turn is discounted by one step before receiving terminal-outcome fill. This form makes the two sources of credit explicit: $c^{(K)}_{g, k}$ captures local answer-likelihood progress, while the terminal fill only affects turns whose look-ahead window reaches the final outcome. The endpoint telescoping property above applies exactly to the one-step component $\delta_k$; the $K$-step backup and terminal fill intentionally trade this exact endpoint-only form for delayed credit propagation and outcome anchoring.

Joint optimization with outcome reward.

We do not group-normalize the turn values. Instead, the TD-derived turn credit is used directly as a trajectory-local auxiliary signal and is jointly optimized with the GRPO outcome reward. For the outcome component, we use the standard group-relative normalization

$ A^{\mathrm{out}}g= \begin{cases} (R_g-\bar{R})/\sigma_R, & \sigma_R>0, \ 0, & \sigma_R=0, \end{cases} \quad \bar{R}=G^{-1}\sum{h=1}^{G}R_h, \quad \sigma_R=\sqrt{G^{-1}\sum_{h=1}^{G}(R_h-\bar{R})^2}. $

For each token generated as part of tool-interaction action $a_{g, k+1}$, we set $\mathrm{turn}(t)=k$; final-answer tokens use the outcome component with the answer-tail weighting described in Appendix A.1. The mixed per-token advantage for tool-interaction tokens is

$ \hat{A}{g, t} =\alpha{\mathrm{out}}, A^{\mathrm{out}}g +\alpha{\mathrm{turn}}, r^{\mathrm{turn}}_{g, , \mathrm{turn}(t)},\tag{11} $

where $\alpha_{\mathrm{out}}, \alpha_{\mathrm{turn}}\ge 0$ control the relative strength of terminal correctness and turn-level credit. Let $\mathcal{I}g$ be the assistant-token indices used in the policy-gradient loss for rollout $g$, and let $\rho{g, t}(\theta)=\pi_\theta(a_{g, t}\mid s_{g, t})/ \pi_{\mathrm{old}}(a_{g, t}\mid s_{g, t})$, where $\pi_{\mathrm{old}}$ is the frozen behavior-policy snapshot that generated the current rollout batch. We optimize the clipped GRPO objective

$ \mathcal{J}{\mathrm{TRACE}}(\theta) =\frac{1}{G}\sum{g=1}^{G}\frac{1}{|\mathcal{I}g|} \sum{t\in\mathcal{I}g} \min!\left(\rho{g, t}(\theta)\hat{A}{g, t}, \operatorname{clip}!\left(\rho{g, t}(\theta), 1-c_-, 1+c_+\right)\hat{A}_{g, t} \right),\tag{12} $

which keeps the standard GRPO group-relative outcome signal while adding dense turn-level credit without normalizing the turn values across the prompt group.

4. Experiments

Section Summary: In these experiments, the authors trained AI search agents on custom multi-document retrieval tasks that require multiple tool calls over an offline corpus, testing whether fine-grained credit assignment improves long-horizon performance when only the final answer receives a correctness reward. They ran controlled reinforcement-learning trials with two Qwen3 model sizes inside a ReAct-style browser interface, comparing their TRACE method against several RL baselines that rely solely on outcome rewards as well as external deep-research systems. Results were measured on a closed-corpus benchmark matching the training setup plus three open-web evaluations to check whether gains transfer across retrieval environments and languages.

4.1 Experimental Setup

Training datasets.

A central question in this work is whether dense turn-level credit can improve long-horizon tool use when the only verifiable supervision is final-answer correctness. To examine this question in a controlled setting, we train agents on synthetic multi-document search tasks built over the offline corpus released by OpenResearcher ([23]). Standard multi-hop QA benchmarks are often too short for this purpose: many examples can be solved with one search and one document open, leaving little room to study delayed credit from later evidence gathering. Our training data instead requires chained retrieval over multiple irreplaceable evidence documents, so early queries, document-selection decisions, and in-document localization steps can all affect the final answer. Appendix A.4 describes the data generation, filtering, and verification pipeline.

Agentic harness.

The policy is trained in a ReAct-style reasoning-and-acting harness ([1]), where each assistant turn contains private reasoning and either one browser action or a final answer. The closed-corpus harness serves documents from the OpenResearcher offline corpus through a FAISS retrieval index built with Qwen3-Embedding-8B ([24]). The browser interface exposes three actions, browser.search, browser.open, and browser.find; the final response must be emitted inside <answer> tags. The outcome reward checks normalized exact-match answer correctness, with a small format component for well-formed answer blocks. Appendix A.3 gives the full tool schema, rollout stopping rules, masking convention, and prompt template.

Models.

To evaluate whether the same credit-assignment signal works across model scale, we instantiate the search policy with two Qwen3 thinking backbones: Qwen3-4B-Thinking-2507 and Qwen3-30B-A3B-Thinking-2507 ([25]). The 4B model is our primary controlled setting for comparing optimization objectives and ablations, while the 30B-A3B model tests whether $\textsc{TRACE}$ transfers to a larger agentic backbone. Unless otherwise stated, both models use the same browser action space, rollout protocol, terminal reward, training data, and evaluation interface. Importantly, all controlled RL runs start directly from the base search policy without a cold-start supervised fine-tuning stage, a mid-training stage, or training on live-web data.

Training protocol.

We optimize each student with Adam at a constant learning rate of $10^{-6}$, using a global batch size of $128$ and $8$ rollouts per prompt, with up to $60$ tool turns per trajectory. $\textsc{TRACE}$ computes turn credit with the log-ratio formulation in Equation 7, using training gap offset $\epsilon_{\mathrm{train}}=10^{-1}$. The look-ahead advantage in Equation 10 uses horizon $K=3$ and discount $\gamma_{\mathrm{td}}=0.8$, with terminal-outcome fill enabled at scale $2.0$. The final per-token advantage in Equation 11 mixes outcome and turn signals with $\alpha_{\mathrm{out}}=1.0$ and $\alpha_{\mathrm{turn}}=0.2$; the token-level distillation term is disabled in this work. Tokens in the final answer span are weighted by the group-mean rule at scale $0.05$. Appendix A.1 lists the full script-level training, rollout, and turn-credit hyperparameters.

Baselines.

We compare $\textsc{TRACE}$ with both external deep-research agents and controlled RL baselines. The external agents include ASearcher-QwQ-32B ([26]), WebDancer-32B ([27]), CutBill-30B-A3B ([28]), and TongyiDS-30B-A3B ([29]); these systems provide strong reference points for long-horizon search, although they may use different data, model backbones, training pipelines, and benchmark harnesses, so we treat their reported numbers as non-controlled reference points. The controlled baselines are initialized from the same Qwen3 backbones as $\textsc{TRACE}$. Base denotes the untrained search policy. GRPO ([10]) uses only the terminal outcome reward, GSPO ([30]) replaces token-level importance ratios with sequence-level ratios, and GiGRPO ([31]) applies group-in-group policy optimization for agent training. For all controlled baselines, we keep the browser action space, rollout protocol, training data, terminal reward, and evaluation interface fixed, so performance differences primarily reflect the optimization objective and credit-assignment signal.

Evaluation.

We evaluate deep-research ability in both closed-web and open-web settings. The closed-web benchmark is BrowseComp-Plus ([22]), for which we build an offline search engine from the officially released corpus using a Qwen3-Embedding-8B ([24]) FAISS index. This setting measures long-horizon reasoning under a reproducible retrieval environment that matches the training interface. To test transfer beyond the closed training corpus, we also evaluate on three open-web benchmarks: BrowseComp ([32]), GAIA ([33]), and xbench-DeepSearch ([34]), where retrieval is served by the Serper API ([35]). Among them, xbench-DeepSearch is a Chinese QA benchmark, allowing us to test whether a training recipe learned in the closed search setting transfers across languages as well as across retrieval environments. Across all evaluations, rollouts stop when the model emits a final <answer> span, reaches the tool-turn limit, or hits a context or generation limit. Unless otherwise stated, controlled ablations are single training runs, so small differences should be read as directional rather than as variance-adjusted conclusions.

4.2 Main Results

::: {caption="Table 1: Performance comparison on long-horizon deep-research benchmarks. BrowseComp-Plus is evaluated in the closed-web setting, while BrowseComp, GAIA, and xbench-DeepSearch are evaluated with open-web retrieval. Avg reports the unweighted average over all four benchmarks. The top block lists external deep-research agents for reference, while the Qwen3 blocks compare controlled RL variants initialized from the same backbone and trained with the same browser interface, data, terminal reward, and evaluation protocol. Highlighted rows denote TRACE."}

:::

TRACE remarkably improves base-model tool use and transfers beyond the training corpus.

Table 1 shows that $\textsc{TRACE}$ substantially improves base-model tool use while retaining generalization beyond the training environment. The most direct evidence comes from the closed-web BrowseComp-Plus setting, where $\textsc{TRACE}$ raises Qwen3-4B from $7.2$ to $35.6$ and Qwen3-30B-A3B from $8.4$ to $42.6$ using only RL with outcome-level and turn-level rewards. This is a strong gain for a minimal recipe: the models are not first adapted with cold-start SFT, agentic mid-training, or live-web data, so the improvement suggests that the base backbones already contain useful reasoning and tool-use primitives, but need a denser credit signal to organize them into effective long-horizon behavior. The external-agent comparisons give the result additional context. Although those systems may use different data and training pipelines, the 4B $\textsc{TRACE}$ agent already surpasses several larger deep-research agents on BrowseComp-Plus, while the 30B-A3B agent approaches the strong TongyiDS-30B-A3B result. The more important controlled test is whether this behavior transfers beyond the closed training corpus: despite being trained in a closed synthetic search environment, $\textsc{TRACE}$ improves both backbones on open-web benchmarks served by an external search API; with Qwen3-30B-A3B, it reaches $12.9$ on BrowseComp, $52.0$ on GAIA, and $45.0$ on the Chinese QA benchmark xbench-DeepSearch. This pattern suggests that $\textsc{TRACE}$ is not merely fitting the closed retrieval corpus within our controlled setup; it teaches a more general interaction strategy for searching, reading observations, and refining later actions across different corpora and languages, though it does not close the gap to the strongest external deep-research system in Table 1.

TRACE outperforms other RL methods across model scales.

The controlled RL comparison clarifies why the gains come from credit assignment rather than from a change in model, data, or environment. All RL variants use the same backbone, browser interface, training data, terminal reward, and evaluation protocol; what differs is how the policy-gradient signal is constructed. Against outcome-only GRPO, $\textsc{TRACE}$ asks the cleanest question: if the final correctness reward is unchanged, does adding turn-level TD credit help? The answer is yes on both scales, with the 4B four-benchmark average rising to $34.0$ from GRPO's $29.5$ and the 30B-A3B average rising to $38.1$ from $32.5$. This indicates that terminal rewards alone do not expose enough information about which parts of a long trajectory should be reinforced. GSPO and GiGRPO address a different aspect of RL optimization by changing sequence- or group-level estimation, but they still operate primarily at the trajectory level: they can compare rollouts, yet they do not assign credit to the intermediate interaction that made later success more likely. This distinction explains why $\textsc{TRACE}$ is especially beneficial on BrowseComp-Plus and xbench-DeepSearch, where success depends on a chain of dependent environment interactions rather than a single decisive final answer. By keeping outcome correctness as the global anchor and adding temporally aligned progress credit within the trajectory, $\textsc{TRACE}$ supplies the kind of supervision that long-horizon agent training is missing.

4.3 Learning Dynamics of TRACE

**Figure 3:** **Learning dynamics on BrowseComp-Plus.** (a) Training reward on Qwen3-4B. (b) Evaluation accuracy on Qwen3-4B. (c) Training reward on Qwen3-30B-A3B. (d) Evaluation accuracy on Qwen3-30B-A3B.

TRACE makes the model converge faster and to a better policy.

Figure 3 compares the learning dynamics of $\textsc{TRACE}$ and outcome-reward baselines across four panels. The first visible trend is that all methods improve as training proceeds, but the $\textsc{TRACE}$ curves start rising earlier. The second trend is that the slope of $\textsc{TRACE}$ is steeper in the early stage, indicating faster reward acquisition rather than only a late-stage separation. The third trend is that the gap does not disappear after the initial rise: $\textsc{TRACE}$ stabilizes at a higher plateau, suggesting that the faster optimization also leads to a better learned policy. Finally, the same ordering appears in held-out accuracy on BrowseComp-Plus with Qwen3-30B-A3B, where the $160$-step $\textsc{TRACE}$ checkpoint already exceeds the $200$-step outcome-reward baseline. These observations suggest that the reward-curve improvement reflects earlier acquisition of transferable long-horizon search behavior. A likely reason is that outcome-only training assigns one terminal signal to the whole trajectory, so early failed rollouts can update useful intermediate progress together with the final mistake. $\textsc{TRACE}$ reduces this temporal mismatch by assigning turn-level TD credit when an interaction makes the gold answer more predictable under the reference model, while still keeping the outcome reward in the objective to enforce final correctness.

TRACE scales up interaction length faster in the beginning stage.

$**Figure 4:** Trajectory-scale dynamics on BrowseComp-Plus with Qwen3-4B. The $\textsc{TRACE}$ curve increases earlier and faster than GRPO, indicating that the two training objectives expand interaction length at different rates during the early training stage.$

Figure 4 shows a clear difference in trajectory scale between $\textsc{TRACE}$ and GRPO on the Qwen3-4B backbone: the $\textsc{TRACE}$ curve starts to rise earlier and grows faster, while the GRPO curve increases more slowly. This suggests that the objectives differ in how they encourage longer environment interaction during early training. With outcome-only GRPO, longer early trajectories often still fail at the final answer, so the whole rollout receives low or negative relative advantage and the objective cannot separate unhelpful extra interaction from intermediate progress. $\textsc{TRACE}$ provides a more local signal: when an interaction and its observation increase answer readiness, the turn-level TD credit can reward that progress before the final answer is correct. This gives a smoother incentive for scaling interaction length, while the outcome reward keeps optimization tied to final correctness.

4.4 Ablation Study

**Figure 5:** **Ablations on BrowseComp-Plus with Qwen3-4B.** (a) Turn-level reward coefficient. (b) TD look-ahead horizon $K$. (c) Reference checkpoint for answer-readiness scoring.

Log-ratio credit assignment is more effective in this run.

::: {caption="Table 2: Credit-assignment ablation on Qwen3-4B."}

:::

We ablate the turn-level credit format on BrowseComp-Plus while keeping the same Qwen3-4B training setup. Starting from outcome-only GRPO ($30.0$), adding dense transition rewards based on the raw log-probability delta improves the score to $32.4$, and normalizing by the remaining answer-likelihood gap further improves it to $34.6$. The proposed log-ratio TD credit achieves the best score in this run ($35.5$), suggesting that relative gap closure provides a more effective credit signal than absolute likelihood changes. This supports the design in Equation 7: the log-ratio form better normalizes credit across states with different confidence levels and preserves a telescoping structure that discourages redundant trajectory extension.

Ablation on turn-level reward coefficient.

Figure 5(a) shows that the weight assigned to the dense turn-level signal must be chosen moderately. A small coefficient underuses the extra supervision from prefix-level progress, while an overly large coefficient can make local reference-model readiness dominate the final correctness signal. In the coefficient sweep, performance rises from $33.6$ to $35.6$ as the coefficient increases from $1$ to $3$, but then drops to $34.5$ and $31.1$ at larger values. This pattern supports the use of turn-level reward as an auxiliary credit signal rather than a replacement for the outcome reward.

Ablation on TD look-ahead horizon $K$.

Figure 5(b) varies the TD look-ahead horizon $K$, which controls the strength of the propagated transition-level signal. The $K=0$ ablation denotes disabling the dense TD backup entirely rather than passing $K=0$ into Equation 9; it obtains only $30.0$ accuracy, close to outcome-only GRPO, because early search and document-selection actions receive little useful credit. Moderate propagation substantially improves performance, reaching $34.7$ and $35.6$, which indicates that delayed evidence from later open and find operations should be assigned back to the preceding retrieval decisions. However, the largest setting reduces accuracy to $28.9$, suggesting that overemphasizing the propagated signal introduces noise from loosely related later turns and weakens the temporal locality of the reward.

Ablation on reference model for probability estimation.

Finally, Figure 5(c) studies which reference checkpoint should be used to compute the gold-answer prefix probabilities. The baseline without reference-model progress scoring reaches $30.0$, while using the initialization checkpoint at step $0$ raises accuracy to $35.6$. Replacing it with a slightly updated checkpoint at step $200$ obtains a similar score of $36.1$ in this run. The small gap between the two reference checkpoints suggests that $\textsc{TRACE}$ does not rely on a specially strong or heavily tuned reference model. Instead, the reference model mainly serves as a stable value-estimation anchor: it provides prefix-level answer-readiness scores from which relative TD progress can be computed. In practice, the base checkpoint is already sufficient for this role, which makes the method easier to apply and reduces dependence on selecting an advanced teacher or a carefully adapted scoring model.

5. Related Work

Section Summary: Recent work has shown that fine-tuning large language models with reinforcement learning and automatic correctness checks can turn them into stronger problem solvers, whether they are answering math questions or using tools like web browsers and code interpreters over multiple steps. A key difficulty in these long interactions is that rewards often arrive only at the end, making it hard to tell which individual decisions helped or hurt. The paper positions its method as a way to give the model clearer feedback on intermediate progress during training, without needing extra learned components or repeated simulations of the same situation.

Post-training of LLMs for reasoning and agentic tasks.

Post-training has become a central recipe for turning pretrained language models into capable problem solvers. For reasoning tasks, reinforcement learning with verifiable rewards has shown that outcome supervision can substantially improve mathematical and symbolic problem solving when correctness can be checked automatically ([9, 10]). These methods typically optimize a trajectory-level signal with clipped policy-gradient updates ([36]): a complete solution receives positive or negative feedback according to whether the final answer is correct. This simple regime is effective for short-horizon reasoning, but becomes less informative as decisions grow into multi-hop or browsing trajectories ([21, 32, 22]). LLM post-training has also been extended from single-response reasoning to agentic settings in which models interact with tools, environments, users, or external feedback over multiple turns. Early systems such as WebGPT and ReAct introduced the interleaving of language-model reasoning, actions, and observations ([3, 1]). Subsequent work improves tool use through self-supervised tool-call prediction, instruction tuning on tool-use traces, and agent-specific tuning corpora ([2, 37, 38]), while related approaches use feedback across trials to revise plans, memory, or behavior ([39]). This agentic post-training landscape now covers a wide range of tasks beyond search. Representative settings include shopping and web navigation ([4, 5, 6, 40, 41, 42]), API and tool use ([2, 37, 43]), software engineering agents and related coding systems, real-computer and mobile-device control ([8, 44, 45]), and multi-turn user-interactive tool use ([46, 47, 48, 49]). Within software engineering, representative work spans role-based multi-agent development and interactive code execution ([50, 51, 52, 53]); repository-level evaluation, planning, and repair ([7, 54, 55, 56, 57, 58, 59, 60]); and executable training environments, scalable task generation, and RL or inference-scaling recipes ([61, 62, 63, 64, 65, 66]). Search and deep-research agents form one important branch of this broader agentic post-training trend, with recent work applying outcome-based RL, synthetic trajectory construction, and tool-use training to search-augmented reasoning and deep browsing ([67, 68, 69, 70, 71, 72, 73, 27, 74, 75, 23, 76, 26]). Recent surveys and position papers summarize these developments under deep search, tool-use agents, and agentic deep research ([77, 78]). Our work is in this general post-training regime: rather than designing a new agent interface or adding a stronger teacher, we study how to make the training signal more informative inside a multi-turn trajectory.

Credit assignment for long-horizon language agents.

Credit assignment is a long-standing challenge in reinforcement learning when the reward is delayed until the end of an extended trajectory ([11]). Temporal-difference learning addresses this problem by estimating how each transition changes the value of the current state ([20]). Potential-based reward shaping similarly uses differences in a state potential to provide denser feedback while preserving the underlying task objective under standard assumptions ([79]), and return-decomposition methods such as RUDDER redistribute delayed rewards to the decisions that make the final outcome predictable ([12]). In language-model reasoning, process supervision and process reward models provide fine-grained feedback by labeling or modeling intermediate reasoning steps ([15, 16, 80]). These approaches motivate dense supervision, but they often require a learned critic, human or synthetic step labels, process-reward annotation, Monte Carlo continuations, or task-specific reward features. Recent work has begun to adapt fine-grained credit assignment to language-model policies. For single-response reasoning, [18] formalize a process reward as the change in the probability of eventually reaching a correct answer, [17] estimate per-step values from Monte Carlo rollouts, and [19] derive an implicit process reward from the log-likelihood ratio between a policy and a fixed reference model. For multi-turn agents, recent work studies stepwise rewards, hindsight credit, memory-aware credit assignment, search-agent RL, and training infrastructure for long-horizon tool use ([31, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 48, 47, 46, 49]). $\textsc{TRACE}$ shares the goal of assigning credit below the trajectory level, but avoids training an additional critic or progress estimator, does not require Monte Carlo continuations, and does not depend on revisiting identical states. At a high level, $\textsc{TRACE}$ complements outcome-based post-training with turn-level feedback, allowing long-horizon agents to learn from useful intermediate progress rather than only from final success or failure.

6. Limitations

Section Summary: The validation of TRACE focused on search agents that produce short, verifiable answers, where progress could be tracked by how much the correct final response became easier to predict after each step. For agents that must output long, structured results such as code patches or open-ended text, the same progress signal may no longer work reliably. This restricts the method’s current scope even though the basic idea of assigning credit at each turn still applies, leaving richer tasks for future work.

Our empirical validation focuses on long-horizon search agents whose final answers are short and can be compared against a known ground truth. This setting matches the core design of $\textsc{TRACE}$: the frozen reference model estimates the value of a trajectory prefix by measuring how much more predictable the gold answer becomes after each tool interaction. For tasks with compact answers, this provides a simple and stable proxy for progress. However, the same value estimation strategy may be less direct for agentic tasks whose correct outputs are long, structured, or open-ended. For example, a code agent may need to produce a multi-file patch rather than a short answer string, and an open-ended assistant may need to satisfy underspecified user preferences rather than match a single gold response. In such settings, it is unclear whether gold-output log-probability under a frozen reference model is still a reliable state-value proxy.

This limitation does not affect the main claim that turn-level credit can reduce the sparsity of outcome-only agentic RL, but it does bound the current scope of the method. Extending $\textsc{TRACE}$ to richer agent tasks may require alternative state-value targets, such as execution-based progress signals for coding, structured task specifications, or decomposed verifiable subgoals. We leave the design of reliable credit estimators for long-output and open-ended agents to future work.

7. Conclusion

Section Summary: The paper introduces TRACE, a simple way to give an AI agent feedback on individual steps during long, multi-turn tasks without needing a separate judge model or human labels. It does this by comparing a frozen reference model's likelihood of reaching the correct final answer at each tool-use point, turning those comparisons into local rewards that are combined with overall outcome scores. Experiments show this approach improves performance over standard methods across several benchmarks, especially on challenging web-search tasks, though the authors note it is currently suited mainly to problems with short, verifiable answers.

We presented $\textsc{TRACE}$, a critic-free method for assigning turn-level credit in long-horizon agentic RL. At tool-call boundaries, a frozen reference model maps gold-answer log-probabilities to log-ratio state values, whose TD changes provide local rewards. Combining this signal with the GRPO outcome advantage preserves final-answer correctness as the training anchor without step labels, a strong LLM judge, or a trained critic or process reward model. $\textsc{TRACE}$ outperforms outcome-only GRPO on all four benchmarks at both model scales, raising the four-benchmark average from $29.5$ to $34.0$ for Qwen3-4B and from $32.5$ to $38.1$ for Qwen3-30B-A3B. On BrowseComp-Plus, it improves the base policies from $7.2$ to $35.6$ and from $8.4$ to $42.6$, with gains transferring to open-web evaluation. These results establish frozen-reference turn credit as an effective signal for long-horizon search with compact, verifiable answers; long, structured, or open-ended outputs will require alternative state-value estimators.

8. Acknowledgment

Section Summary: The authors thank Xuhui Huang for his helpful feedback on the project. They also note that the research received partial funding from several sources, including a University of Wisconsin-Madison research grant, awards from the Air Force Office of Scientific Research and the National Science Foundation, support from the Office of Naval Research, Schmidt Sciences, Open Philanthropy, and an Alfred P. Sloan Fellowship.

We gratefully acknowledge Xuhui Huang for his valuable feedback. The work is supported in part by the Research Forward Fund from the University of Wisconsin-Madison Office of the Vice Chancellor for Research, AFOSR Young Investigator Program under award number FA9550-23-1-0184, National Science Foundation under awards IIS-2237037 and IIS-2331669, Office of Naval Research, Schmidt Sciences Foundation, Open Philanthropy (now Coefficient Giving), and Alfred P. Sloan Fellowship.

Appendix

Section Summary: The appendix lists the full set of training, rollout, evaluation, and optimization settings used for the Search-R1 run, along with the extra reward-weight and turn-credit values that activate the TRACE method. It then explains why a log-ratio formulation of remaining answer-likelihood gap is preferred over raw log-probability differences or simple linear normalization, because the ratio-based TD reward both measures relative progress and telescopes to depend only on the initial and final states. A comparison table highlights how each approach behaves with respect to order preservation, scaling, and numerical stability.

A.1 Training Hyperparameters

Table 3 and Table 4 summarize the script-level hyperparameters used for the Qwen3-30B-A3B-Thinking-2507 Search-R1 training run. The shared launch script keeps turn-level scoring disabled by default for a single-node colocated layout unless a remote reference-model scoring endpoint is provided; $\textsc{TRACE}$ runs enable this path and use the settings in Table 4.

\begin{tabular}{@p{0.18\linewidth}p{0.33\linewidth}p{0.41\linewidth}@}
\toprule
{\bfseries Category} & {\bfseries Hyperparameter} & {\bfseries Value} \\
\midrule
\multirow{14}{*}{Rollout}
     {} & Rollout engine & SGLang \\
     {} & GPUs per rollout engine & $1$ \\
     {} & Running requests & $64$ \\
     {} & Rollout batch size & $64$ \\
     {} & Samples per prompt & $8$ \\
     {} & Sampling temperature & $1.0$ \\
     {} & Sampling top- $p$ & $1.0$ \\
     {} & Max trajectory length & $48{,}000$ tokens \\
     {} & Max response tokens & $4096$ tokens \\
     {} & Max tool turns & $60$ \\
     {} & Max observation tokens & $2048$ tokens \\
     {} & Rollout timeout & $240$ s \\
\midrule
\multirow{4}{*}{Evaluation}
     {} & Eval samples per prompt & $1$ \\
     {} & Eval max response length & $8192$ tokens \\
     {} & Eval max context length & $72{,}000$ tokens \\
     {} & Eval max tool turns & $80$ tool turns \\
\midrule
\multirow{14}{*}{Optimization}
     {} & Global batch size & $128$ \\
     {} & Optimizer & Adam \\
     {} & Learning rate schedule & Constant \\
     {} & Learning rate & $1\times 10^{-6}$ \\
     {} & Weight decay & $0.01$ \\
     {} & Adam $\beta_1$ & $0.9$ \\
     {} & Adam $\beta_2$ & $0.98$ \\
     {} & Optimizer offload & CPU offload with overlapped D2H/H2D transfer \\
     {} & KL loss & $0$ \\
     {} & Entropy coefficient & $0.00$ \\
     {} & Clip ratio lower bound & $0.20$ \\
     {} & Clip ratio upper bound & $0.28$ \\
     {} & Loss accounting & Per-token loss \\
     {} & TIS & Enabled \\
\bottomrule
\end{tabular}

\begin{tabular}{@p{0.20\linewidth}p{0.36\linewidth}p{0.36\linewidth}@}
\toprule
{\bfseries Category} & {\bfseries Hyperparameter} & {\bfseries Value} \\
\midrule
\multirow{3}{*}{Outcome reward}
     {} & Outcome reward coefficient & $1.0$ \\
     {} & Format score & $0.1$ \\
     {} & Zero-advantage handling & Return zero when the group advantage is zero \\
\midrule
\multirow{19}{*}{Turn credit}
     {} & Turn-reward scoring & Enabled for \textsc{TRACE} runs with a remote reference-model scoring endpoint \\
     {} & Turn-reward weight & $0.2$ \\
     {} & Training gap offset $\epsilon_{\mathrm{train}}$ & $0.1$ \\
     {} & Ground-truth field & \texttt{ground\_truth} \\
     {} & Advantage mode & \texttt{turn\_advantages} \\
     {} & Remaining-space transform & Enabled \\
     {} & Transform mode & \texttt{log\_ratio} \\
     {} & Legacy transform stabilizer & $0.001$ \\
     {} & TD look-ahead horizon & $K=3$ \\
     {} & TD discount & $\gamma_{\mathrm{td}}=0.8$ \\
     {} & $K$-step normalization & Enabled \\
     {} & Terminal-outcome fill & Enabled \\
     {} & Terminal-outcome scale & $2.0$ \\
     {} & Advantage normalization & \texttt{none} \\
     {} & Post-group normalization & False \\
     {} & Advantage clipping & $0.0$ \\
\bottomrule
\end{tabular}

A.2 Comparison with Raw Log-Probability Differences

This section expands on the choice of the log-ratio state value in Equation 6 and the resulting TD action reward in Equation 7. For a fixed rollout, write $\bar{\ell}_k$ for the raw average gold-answer log-probability and $d_k=-\bar{\ell}_k+\epsilon$ for the remaining answer-likelihood gap. The raw log-probability difference $\Delta^{\ell}k=\bar{\ell}{k+1}-\bar{\ell}_k$ telescopes to $\bar{\ell}_T-\bar{\ell}0$ and therefore preserves the ordering of two rollouts from the same prompt when they share the same initial state. However, raw differences measure absolute improvement in log-probability, not the fraction of the remaining gap that a turn closes. The linear remaining-gap score $\widetilde{\Delta}^{\mathrm{lin}}k=(\bar{\ell}{k+1}-\bar{\ell}k)/d_k$ and the TD reward from the log-ratio value, $C_k=V{k+1}-V_k=\log(d_k/d{k+1})$, both correct this scale issue by rewarding relative gap reduction, but only the log-ratio value also retains an endpoint-only telescoping form.

\begin{tabular}{@p{0.22\linewidth}p{0.22\linewidth}p{0.24\linewidth}p{0.22\linewidth}@}
\toprule
Property & Raw delta $\Delta^{\ell}_k$ & Linear remaining gap $\widetilde{\Delta}^{\mathrm{lin}}_k$ & Log-ratio TD reward $C_k$ \\
\midrule
Definition & $\bar{\ell}_{k+1}-\bar{\ell}_k$ & $(\bar{\ell}_{k+1}-\bar{\ell}_k)/d_k$ & $V_{k+1}-V_k=\log(d_k/d_{k+1})$ \\
Telescoping & $\bar{\ell}_T-\bar{\ell}_0$ & No endpoint-only form & $V_T-V_0=\log(d_0/d_T)$ \\
Order preservation & Yes, for shared $\bar{\ell}_0$ & Not guaranteed & Yes, for shared $\bar{\ell}_0$ \\
Per-step unit & Absolute improvement & Relative gap reduction & Relative gap reduction \\
Large remaining gap & Uniform scaling & Downweights early large-gap steps & Downweights early large-gap steps \\
Small remaining gap & Uniform scaling & Strongly amplifies near-terminal steps & Moderately amplifies near-terminal steps \\
Numerical behavior & Stable & Can spike when $d_k$ is small & Finite with offset and logarithmic growth \\
\bottomrule
\end{tabular}

An offline diagnostic on a held-out BrowseComp-Plus evaluation trace further supports this choice. We compared the cumulative score from each formulation against the final reference-model log-probability and the binary outcome reward over $830$ rollouts containing $3742$ tool turns, using a separate diagnostic offset $\epsilon_{\mathrm{diag}}=10^{-3}$ for the two remaining-gap normalizations. The log-ratio score has the strongest correlation with both endpoint quality and final correctness, while also giving the best pairwise ranking accuracy among rollouts with similar initial raw log-probabilities.

\begin{tabular}{@lrrr@}
\toprule
Metric & Raw & Linear & Log-ratio \\
\midrule
Correlation with final $\bar{\ell}_T$ & $0.425$ & $0.721$ & $\mathbf{0.751}$ \\
Correlation with positive outcome reward & $0.603$ & $0.680$ & $\mathbf{0.713}$ \\
Pairwise ranking accuracy, $|\Delta \bar{\ell}_0|\le 0.2$ & $97.34\%$ & $93.13\%$ & $\mathbf{98.24\%}$ \\
\bottomrule
\end{tabular}

The difference is visible at the turn level. Consider two transitions with nearly identical raw gains: one moves from $\bar{\ell}=-5.1187$ to $\bar{\ell}=-1.5712$ ($\Delta^{\ell}=3.5475$), while another moves from $\bar{\ell}=-10.6570$ to $\bar{\ell}=-7.1061$ ($\Delta^{\ell}=3.5509$). Raw delta treats them as essentially equal. In contrast, the corresponding log-ratio TD rewards are $1.1806$ and $0.4052$, respectively, because the first transition closes a much larger fraction of its remaining gap. This behavior is desirable for long-horizon search: decisive evidence localization near the answer should receive more credit than an equally sized absolute improvement that still leaves the rollout far from predicting the gold answer.

A.3 Search-Agent Task and Interface

We instantiate $\textsc{TRACE}$ on a deliberately simple closed-corpus search task. Given a question $x$, the agent follows an interleaved reasoning-and-acting pattern ([1]): each assistant turn contains private reasoning and either one browser tool call or a final answer. If the turn contains a valid tool call, the environment executes it against the retrieval corpus and appends the resulting observation before the next assistant turn. Keeping this interface small helps isolate the effect of the training signal from improvements due to a more specialized planner or tool stack.

Rollouts stop when the model emits a final <answer> span, reaches the tool-turn limit, or hits a context or generation limit. Training loss is applied only to assistant-generated tokens, while tool observations are masked. The outcome reward checks whether the normalized final answer exactly matches the gold answer, with a small format component for well-formed answers. We also record lightweight trajectory diagnostics, including tool-call counts, repeated queries, reopened documents, repeated in-document searches, navigation errors, and answer-format validity.

The environment exposes three browsing tools over the retrieval corpus. Table 7 summarizes the tool schema used in all experiments.

\begin{tabular}{lll p{0.39\linewidth}}
\toprule
Tool & Argument & Type/default & Description \\
\midrule
\texttt{browser.search} & \texttt{query} & string, required & Search query string. \\
  {} & \texttt{topn} & integer, default $10$ & Number of top results to return. \\
\midrule
\texttt{browser.open} & \texttt{id} & integer, default $-1$ & Result index from the most recent search, using zero-based indexing; $-1$ rereads the current page. \\
  {} & \texttt{loc} & integer, default $-1$ & Starting line number; $-1$ uses the beginning of the page or a relevant passage. \\
  {} & \texttt{num\_lines} & integer, default $-1$ & Number of lines to display; $-1$ uses the environment default. \\
\midrule
\texttt{browser.find} & \texttt{pattern} & string, required & Exact text pattern to search for in the currently open document. \\
  {} & \texttt{cursor} & integer, default $-1$ & Page cursor to search in; $-1$ searches the current page. \\
\bottomrule
\end{tabular}

The assistant's reasoning, tool call, and final-answer formatting tokens are part of the assistant turn for training purposes, while tool observations are environment outputs and remain masked from the policy-gradient loss.

A.4 Synthetic Training Data

Training a search agent with verifiable reinforcement learning requires questions whose rewards remain informative beyond the single-hop retrieval regime. If the training distribution is dominated by direct lookup questions, the agent quickly saturates: once it can issue one or two effective searches, additional training provides little pressure to learn deeper information gathering or cross-document synthesis. We therefore construct a synthetic training set of hard multi-document identification questions designed to require chained search, reasoning over at least two irreplaceable source documents, and resistance to trivial answer lookup.

A.4.1 Generation Pipeline

The pipeline starts from a document corpus, stored as parquet files or a Hugging Face dataset, together with a related-document index in JSONL format. For each anchor document, we retrieve the top- $K_{\mathrm{rel}}$ related documents whose similarity score exceeds a minimum threshold and use this set as the source context for question generation. This related-document context increases the probability that a valid question can connect facts across documents while still requiring the model to identify which documents are necessary.

For every anchor document, we run a best-of- $B$ generation procedure with rejection sampling. Each independent candidate is generated by an LLM prompted to produce a hard question, its answer, and an explicit reasoning chain over the provided sources. If a candidate fails verification, the rejection reason is fed back to the generator for up to a fixed number of regeneration attempts. Candidates that pass the initial checks are then sent to an independent answer verification stage: a separate LLM call receives only the question and source documents, not the proposed answer, and must derive the answer from scratch. We discard the example unless the independently derived answer matches the proposed answer under fuzzy matching. Among the remaining candidates, we keep the highest-scoring example and write it to the training JSONL.

A.4.2 Question Templates

The generator is constrained to produce questions from a small set of templates that naturally require cross-document reasoning. Table 8 summarizes the templates used in our data synthesis pipeline.

: Table 8: Question templates used to synthesize hard multi-document QA examples.

Type	Description
`BRIDGE_ENTITY`	Identify a masked entity, then retrieve one of its attributes from another document.
`INTERSECTION`	Find the unique entity satisfying independent conditions from different documents.
`COUNTING_FILTERED`	Count members of a set defined by cross-document criteria.
`COMPARATIVE`	Compare obfuscated entities using facts distributed across documents.
`REVERSE_LOOKUP`	Given an obfuscated output, identify its creator or source through another document.

To prevent direct keyword lookup, each question applies at least one obfuscation operation. Time expressions may be blurred into relative or approximate descriptions; entity names may be masked; names may be replaced by role descriptions; and quantitative constraints may be converted into qualitative ones. These transformations force the agent to first resolve what should be searched for and only then retrieve the evidence needed to answer.

A.4.3 Verification and Selection

Quality control combines programmatic checks with LLM-based verification. The programmatic check requires the reasoning chain to cite at least two distinct source URLs. The LLM verifier then evaluates whether the question is actually hard, grounded, and multi-document. It rejects candidates whose answer appears directly in the question, whose constraints can be satisfied from a single source, whose answer is retrievable with a trivial query, or whose reasoning chain lacks a genuine synthesis step. It also filters unsupported or hallucinated answers and examples in which one of the cited documents is redundant.

The final selection score favors candidates that pass both quality and answer verification, succeed in fewer regeneration attempts, use longer reasoning chains, cite more distinct documents, and assign primary roles to all cited documents. It penalizes questions that leak the answer or produce overly long answers. This scoring rule selects examples that are not merely answerable, but useful for training long-horizon search behavior: the agent must issue multiple queries, maintain intermediate entities, and combine evidence from multiple documents before producing the final answer.

A.5 Qualitative Analysis of the Turn Credit

Outcome-only GRPO assigns one trajectory-level advantage to every action in a rollout, obscuring whether an individual turn retrieves decisive evidence, adds redundant context, or discards an answer already supported by the transcript. We examine five successful and five failed trajectories from rollout batches 0140–0149. In each successful case, we highlight the tool call that closes most of the answer-likelihood gap; in each failed case, the trajectory first reaches an answer-secured prefix and then loses it after one diagnostic tool call. Each example reports the complete question and answer, the one-step credit sequence, and the most diagnostic tool call together with its credit.

For a trajectory with $T$ tool calls, search_r1_turn_values contains $T+2$ frozen-reference values: one before tool use, one after each tool observation, and one after answer generation. Larger $V$ indicates greater closure of the gold-answer likelihood gap; we use $V\geq-0.3$ as a diagnostic definition of an answer-secured prefix. Following Equation 7, $\delta_k=V(S_{k+1})-V(S_k)$ is assigned to zero-based tool turn $k$, with the final difference assigned to answer generation. We mark positive progress in green, harmful transitions in red, and near-zero or mixed credit in gray. Tool calls use the notation browser.tool(argument=value); search-relative open IDs are accompanied by the returned page name.

A.5.1 Successful Trajectories: Localizing the Turn That Wins


**Question.**
In the rail-safety material from around the early 2010s, a conference is said
to be opened by a senior infrastructure executive (not named here). Follow
that trail to the police-led plan later described as helping prevent rail
suicides, then move to a later partnership update and identify the person
quoted as managing the suicide-prevention programme for the rail industry. Who
is that person?

**Answer:** *Ian Stevens*.  

**Final answer:**
    $\checkmark$ **Correct**

**Turn-credit distribution.**

$
\delta=[+1.63, -1.03, -1.04, +6.05, +0.03].
$

**Critical turn.**

**Turn 3.** `browser.open(id=2)`
(Samaritans–Network Rail partnership update)

**Turn credit:** $\delta_3=+6.05$.

After two plausible but non-answer-bearing opens, this page names Ian Stevens
and supplies the missing discriminative fact.


**Question.**
Identify the venue where the Nepali boxer was defeated by the Filipino
Olympian in a unanimous decision, if that Filipino was the same athlete once
knocked out in a world-title rematch by the Thai ex-IBF flyweight champion who
made an impressive Rio start around the mid-2010s.

**Answer:** *Jakarta Expo Boxing Hall*.  

**Final answer:**
    $\checkmark$ **Correct**

**Turn-credit distribution.**

$
\delta=[+0.88, +5.86, +0.00, -0.00].
$

**Critical turn.**

**Turn 1.** `browser.open(id=1)`
(Rappler match report)

**Turn credit:** $\delta_1=+5.86$.

The report supplies "at the Jakarta Expo Boxing Hall" and immediately secures
the venue. The subsequent literal `find` receives $+0.00$, illustrating
that repeated confirmation cannot inflate the telescoping signal.

A.5.2 Negative Trajectories: Localizing the Tool Call That Loses

Every failure below reaches an answer-secured prefix and then undergoes a single large negative transition. The examples differ in what triggers the collapse: a query-echo self-match, an over-interpreted no-match, a match in the wrong context, an over-literal type comparison, or a phrase copied from the question rather than read from the page.


**Question.**
In a post from around the mid-2010s by a youth organizer planning an
end-of-season teen church overnight, the writer jokes about a very high-voltage
fence and doubts a loft drop trust exercise. Use that to find another
reflection from roughly the same era where students mention trusting the
rope-holder during a heights course, then compare the fear-focused activity
named there with the similarly fear-themed activity listed in a different
school day-log that also mentions climbing, archery, a blindfold obstacle
course, and a ladder challenge. Which fear-themed activity title comes later
in alphabetical order?

**Gold answer:** *Leap of Faith*.

**Model answer:** *loft drop trust exercise*.

**Final answer:**
    $\times$ **Incorrect**

**Turn-credit distribution.**

$
\delta=[+1.47, +0.13, +1.49, -0.00, -4.83, -0.78].
$

**Critical turn.**

**Turn 4.**
`browser.find(pattern="loft drop trust exercise", cursor=0)`

**Turn credit:** $\delta_4=-4.83$.

After the gold answer is secured, this call matches the model's earlier query
echo rather than a page-supported title. Accepting its own paraphrase as a
second candidate discards the answer and produces the sharp negative credit.


**Question.**
Starting from the article about a TV doctor's health-content partnership
launched on a global blood-disorder awareness observance in the late 2010s,
use that observance date to find another program episode from around the
mid-2010s featuring three women in the same disease space. Then use the
early-2010s Pittsburgh interview with the longtime local nonprofit executive
(described by role, not name) to filter that trio by shared family name. How
many women in the trio match that surname?

**Gold answer:** $2$.  

**Model answer:** $0$.  

**Final answer:**
    $\times$ **Incorrect**

**Turn-credit distribution.**

$
\delta=[+2.31, +0.10, -1.32, +0.09, +0.35, +0.98, -4.05, -1.09].
$

**Critical turn.**

**Turn 6.** `browser.find(pattern="Bush")`
(returns "No matches")

**Turn credit:** $\delta_6=-4.05$.

Although the preceding prefix is answer-secured, the model interprets absence
from an incomplete episode page as evidence that none of the women is a Bush.

The negative credit localizes this invalid factual-negative inference.


**Question.**
Starting from a celebrity-photo page from around the mid-2010s that praises two
aristocratic sisters for attending separate events in England, follow to the
article comparing the elder future queen consort's conservative red charity
gown with her younger sibling's more revealing red look earlier that month.

Using those two sisters as the candidate set, then check a later royal-fashion
tag page from the early 2020s and count how many of those candidates are
explicitly mentioned in the sentence about children serving as page attendants
and bridesmaid at the younger sister's wedding.

**Gold answer:** $2$.  

**Model answer:** $1$.  

**Final answer:**
    $\times$ **Incorrect**

**Turn-credit distribution.**

$
\delta=[+2.19, -0.35, +0.10, -1.03, +0.96, +0.50, -3.63, +0.19, -1.22].
$

**Critical turn.**

**Turn 6.**
`browser.find(pattern="Pippa Middleton")`

**Turn credit:** $\delta_6=-3.63$.

The call succeeds, but matches "as she did for her aunt Pippa Middleton last
year, " a sentence about the wrong wedding. From a just-secured prefix, the
model anchors on this superficially relevant single-name match and later
answers $1$. Tool success is therefore not equivalent to information progress.


**Question.**
Using a post from around the early 2010s by the person who says he ordered
interchangeable raw-brass ring parts as your starting clue, go to a different
instrument-forum thread that references a repair specialist by first name and
mentions that writer considering an instrument with many strings. From that
thread, take the distinct instrument types named there as the candidate set.

Then check a separate dealer page listing "other instrument offerings" from
the mid-2000s era and count how many of those candidate types are present
there.

**Gold answer:** $1$.  

**Model answer:** $0$.  

**Final answer:**
    $\times$ **Incorrect**

**Turn-credit distribution.**

$
\delta=[+1.42, -0.10, +0.82, +0.08, -3.41, -0.97].
$

**Critical turn.**

**Turn 4.** `browser.find(pattern="autoharp")`
(returns "No matches")

**Turn credit:** $\delta_4=-3.41$.

The dealer page has already secured the answer, but the task requires comparing
the candidates by instrument type. The model instead probes one literal string
and treats its absence as proof that the intersection is empty, discarding the
page-level evidence.


**Question.**
Using the workshop notice from around the early 2010s that mentions a big
northern English city and a hotel-branded venue, trace forward to a later
full-day procurement training page to recover a per-attendee cost, then apply
that amount as a strict threshold to the industry-only Canadian pricing
workshops held a few years afterward. How many city sessions in that Canadian
notice have fees greater than that threshold?

**Gold answer:** $2$.  

**Model answer:** $0$.  

**Final answer:**
    $\times$ **Incorrect**

**Turn-credit distribution.**

$
\delta=[+2.00, +0.43, +0.06, -2.46, +0.40, +1.98, -3.48, -1.03].
$

**Critical turn.**

**Turn 6.**
`browser.find(pattern="fees greater than")` (returns "No matches")

**Turn credit:** $\delta_6=-3.48$.

The Canadian workshop page has already supplied the city-level fees. The model
nevertheless searches for a comparison phrase copied from the question rather
than comparing the listed numbers, then over-interprets the no-match and
answers $0$.

Across the positive cases, nearly all credit lands on the search or open that first supplies the missing discriminative fact; redundant probes and post-hoc find calls remain near zero. Across the negative cases, all five trajectories first reach the answer-secured region, after which one misleading find receives the dominant penalty. The calls cover both empty matches that are over-interpreted and successful matches in the wrong context, showing that tool completion alone does not imply progress. Outcome-only GRPO cannot express any of these within-trajectory distinctions because it assigns every turn the same trajectory-level advantage.

References

Section Summary: The references compile dozens of recent academic papers and preprints on language models that combine reasoning with tool use and autonomous action. They emphasize benchmarks for web agents and real-world tasks, reinforcement learning techniques for training, and methods that reward both the process and final results of model behavior. Foundational texts on reinforcement learning sit alongside studies of specific systems such as DeepSeek, Qwen, and various web-interaction environments.

[1] Yao et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations.

[2] Schick et al. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. In Advances in Neural Information Processing Systems.

[3] Nakano et al. (2021). WebGPT: Browser-assisted Question-answering with Human Feedback. arXiv preprint arXiv:2112.09332.

[4] Yao et al. (2022). WebShop: Towards Scalable Real-world Web Interaction with Grounded Language Agents. In Advances in Neural Information Processing Systems.

[5] Deng et al. (2023). Mind2Web: Towards a Generalist Agent for the Web. In Advances in Neural Information Processing Systems.

[6] Zhou et al. (2024). WebArena: A Realistic Web Environment for Building Autonomous Agents. In International Conference on Learning Representations.

[7] Jimenez et al. (2024). SWE-bench: Can Language Models Resolve Real-world GitHub Issues?. In International Conference on Learning Representations.

[8] Xie et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-ended Tasks in Real Computer Environments. arXiv preprint arXiv:2404.07972.

[9] Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300.

[10] Guo et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Nature. 645. pp. 633–638. doi:10.1038/s41586-025-09422-z.

[11] Sutton, Richard S. and Barto, Andrew G. (2018). Reinforcement Learning: An Introduction. MIT Press.

[12] Arjona-Medina et al. (2019). RUDDER: Return Decomposition for Delayed Rewards. In Advances in Neural Information Processing Systems.

[13] Ye et al. (2025). Beyond correctness: Harmonizing process and outcome rewards through RL training. arXiv preprint arXiv:2509.03403.

[14] Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems.

[15] Uesato et al. (2022). Solving Math Word Problems with Process- and Outcome-Based Feedback. arXiv preprint arXiv:2211.14275.

[16] Lightman et al. (2023). Let's Verify Step by Step. arXiv preprint arXiv:2305.20050.

[17] Wang et al. (2024). Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. arXiv preprint arXiv:2312.08935.

[18] Setlur et al. (2024). Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning. arXiv preprint arXiv:2410.08146.

[19] Yuan et al. (2024). Free Process Rewards without Process Labels. arXiv preprint arXiv:2412.01981.

[20] Sutton, Richard S. (1988). Learning to Predict by the Methods of Temporal Differences. Machine Learning. 3. pp. 9–44.

[21] Yang et al. (2018). HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 2369–2380.

[22] Chen et al. (2025). BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent. arXiv preprint arXiv:2508.06600.

[23] Li et al. (2026). OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesis. arXiv preprint arXiv:2603.20278.

[24] Zhang et al. (2025). Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. arXiv preprint arXiv:2506.05176.

[25] Yang et al. (2025). Qwen3 Technical Report. arXiv preprint arXiv:2505.09388.

[26] Gao et al. (2025). Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL. arXiv preprint arXiv:2508.07976.

[27] Wu et al. (2025). WebDancer: Towards Autonomous Information Seeking Agency. arXiv preprint arXiv:2505.22648.

[28] Wu et al. (2025). Cut the Bill, Keep the Turns: Affordable Multi-Turn Search RL. project page.

[29] Tongyi DeepResearch Team et al. (2025). Tongyi DeepResearch Technical Report. arXiv preprint arXiv:2510.24701.

[30] Zheng et al. (2025). Group Sequence Policy Optimization. arXiv preprint arXiv:2507.18071.

[31] Feng et al. (2025). Group-in-Group Policy Optimization for LLM Agent Training. arXiv preprint arXiv:2505.10978.

[32] Wei et al. (2025). BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents. arXiv preprint arXiv:2504.12516.

[33] Mialon et al. (2024). GAIA: A Benchmark for General AI Assistants. In International Conference on Learning Representations.

[34] Chen et al. (2025). xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations. arXiv preprint arXiv:2506.13651.

[35] Serper.dev (2026). Serper: Google Search API. https://serper.dev.

[36] Schulman et al. (2017). Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347.

[37] Qin et al. (2023). ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. arXiv preprint arXiv:2307.16789.

[38] Zeng et al. (2023). AgentTuning: Enabling Generalized Agent Abilities for LLMs. arXiv preprint arXiv:2310.12823.

[39] Shinn et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv preprint arXiv:2303.11366.

[40] Koh et al. (2024). VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. arXiv preprint arXiv:2401.13649.

[41] Drouin et al. (2024). WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?. arXiv preprint arXiv:2403.07718.

[42] Pan et al. (2024). WebCanvas: Benchmarking Web Agents in Online Environments. arXiv preprint arXiv:2406.12373.

[43] Li et al. (2023). API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs. arXiv preprint arXiv:2304.08244.

[44] Rawles et al. (2025). AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. arXiv preprint arXiv:2405.14573.

[45] Zhang et al. (2023). AppAgent: Multimodal Agents as Smartphone Users. arXiv preprint arXiv:2312.13771.

[46] Zhao et al. (2025). MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for Agentic Tool Use. arXiv preprint arXiv:2508.18669.

[47] Chai et al. (2025). RLFactory: A Plug-and-Play Reinforcement Learning Post-Training Framework for LLM Multi-Turn Tool-Use. arXiv preprint arXiv:2509.06980.

[48] Xi et al. (2025). AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning. arXiv preprint arXiv:2509.08755.

[49] Zhang et al. (2025). AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework. arXiv preprint arXiv:2510.04206.

[50] Hong et al. (2024). MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework. In International Conference on Learning Representations.

[51] Qian et al. (2024). ChatDev: Communicative Agents for Software Development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. pp. 15174–15186. doi:10.18653/v1/2024.acl-long.810.

[52] Yang et al. (2023). InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback. In Advances in Neural Information Processing Systems. pp. 23826–23854.

[53] Wang et al. (2024). Executable Code Actions Elicit Better LLM Agents. In International Conference on Machine Learning.

[54] Yang et al. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. arXiv preprint arXiv:2405.15793.

[55] Zhang et al. (2024). CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. pp. 13643–13658. doi:10.18653/v1/2024.acl-long.737.

[56] Bairi et al. (2024). CodePlan: Repository-Level Coding using LLMs and Planning. Proceedings of the ACM on Software Engineering. 1(FSE). pp. 675–698. doi:10.1145/3643757.

[57] Zhang et al. (2024). AutoCodeRover: Autonomous Program Improvement. In Proceedings of the ACM International Symposium on Software Testing and Analysis. pp. 1592–1604. doi:10.1145/3650212.3680384.

[58] Bouzenia et al. (2025). RepairAgent: An Autonomous, LLM-Based Agent for Program Repair. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering. pp. 2188–2200. doi:10.1109/ICSE55347.2025.00157.

[59] Wang et al. (2025). OpenHands: An Open Platform for AI Software Developers as Generalist Agents. In International Conference on Learning Representations.

[60] Xia et al. (2024). Agentless: Demystifying LLM-based Software Engineering Agents. arXiv preprint arXiv:2407.01489.

[61] Pan et al. (2025). Training Software Engineering Agents and Verifiers with SWE-Gym. In International Conference on Machine Learning.

[62] Yang et al. (2025). SWE-smith: Scaling Data for Software Engineering Agents. In Advances in Neural Information Processing Systems.

[63] Wei et al. (2025). SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution. In Advances in Neural Information Processing Systems.

[64] Jain et al. (2025). R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents. arXiv preprint arXiv:2504.07164.

[65] Wang et al. (2025). SWE-Dev: Building Software Engineering Agents with Training and Inference Scaling. In Findings of the Association for Computational Linguistics: ACL 2025.

[66] Mu et al. (2023). LLM-Assisted Incrementality (Uplift) Modeling for Email Advertising: From Feature Interactions to Interpretable Audience–Creative–Channel Policies. Journal of Advanced Computing Systems. 3(1). pp. 31–48.

[67] Li et al. (2025). Search-o1: Agentic Search-Enhanced Large Reasoning Models. arXiv preprint arXiv:2501.05366.

[68] Chen et al. (2025). ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning. arXiv preprint arXiv:2503.19470.

[69] Jin et al. (2025). Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. arXiv preprint arXiv:2503.09516.

[70] Song et al. (2025). R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2503.05592.

[71] Sun et al. (2025). ZeroSearch: Incentivize the Search Capability of LLMs without Searching. arXiv preprint arXiv:2505.04588.

[72] Mu et al. (2025). Offline counterfactual evaluation for advertising and recommendation slot policies: A reproducible study on the open bandit dataset (small). Journal of Technology Informatics and Engineering. 4(3). pp. 521–543.

[73] Li et al. (2025). WebThinker: Empowering Large Reasoning Models with Deep Research Capability. arXiv preprint arXiv:2504.21776.

[74] Li et al. (2025). WebSailor: Navigating Super-human Reasoning for Web Agent. arXiv preprint arXiv:2507.02592.

[75] Zheng et al. (2025). DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments. arXiv preprint arXiv:2504.03160.

[76] Wang et al. (2026). Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modeling. arXiv preprint arXiv:2606.04284.

[77] Xi et al. (2025). A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challenges. arXiv preprint arXiv:2508.05668.

[78] Zhang et al. (2025). From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agents. arXiv preprint arXiv:2506.18959.

[79] Ng et al. (1999). Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. In Proceedings of the Sixteenth International Conference on Machine Learning. pp. 278–287.

[80] Wendi Li and Yixuan Li (2025). Process Reward Model with Q-Value Rankings. In Proceedings of the International Conference on Learning Representations.

[81] Wang et al. (2025). SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution. arXiv preprint arXiv:2505.20732.

[82] Dong et al. (2025). Agentic Reinforced Policy Optimization. arXiv preprint arXiv:2507.19849.

[83] Wang et al. (2025). StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization. arXiv preprint arXiv:2505.15107.

[84] Zhao et al. (2025). R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning. arXiv preprint arXiv:2506.04185.

[85] Wang et al. (2026). Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents. In International Conference on Learning Representations.

[86] Tan et al. (2026). Hindsight Credit Assignment for Long-Horizon LLM Agents. arXiv preprint arXiv:2603.08754.

[87] Yan et al. (2026). Memory-R2: Fair Credit Assignment for Long-Horizon Memory-Augmented LLM Agents. arXiv preprint arXiv:2605.21768.

[88] Qu et al. (2025). Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learning. arXiv preprint arXiv:2412.11120.

[89] Dong et al. (2025). Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning. arXiv preprint arXiv:2505.16410.

[90] Jin et al. (2025). An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents. arXiv preprint arXiv:2505.15117.