From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement
Qinsi Wang $^{1}$ Jing Shi $^{2}$ Huazheng Wang $^{3}$ Kun Wan $^{2}$ Yiran Wu $^{4}$ Bo Liu $^{5}$
Qingyun Wu $^{4}$ Hai Helen Li $^{1}$ Yiran Chen $^{1}$ Handong Zhao $^{2,\dagger}$ Wentian Zhao $^{6,\dagger,\ddagger}$
$^{1}$ Duke University $^{2}$ Adobe Inc. $^{3}$ Oregon State University
$^{4}$ Pennsylvania State University $^{5}$ National University of Singapore $^{6}$ Amazon
$^{\dagger}$Co-advisors.
$^{\ddagger}$Work done at Adobe.
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has driven recent progress in reasoning-oriented large language models (LLMs) by enabling large-scale optimization. However, its applicability remains largely limited to domains such as mathematics and coding, where correctness can be deterministically verifiable. Open-ended tasks instead often rely on human preferences, reward models, or LLM-based judges, introducing evaluation bias, judge capability bottlenecks, and additional inference costs. Drawing on the principle of self-supervised learning, which constructs pretext tasks to derive supervision from the data itself, we propose Reinforcement Learning with Self-Verifiable Rewards (RLSVR), a task-transformation-based training paradigm for extending RLVR to open-ended tasks. RLSVR transforms open-ended tasks into verifiable proxy environments whose internal rules and interaction outcomes automatically generate reward signals. We instantiate RLSVR with SpyRL, a Self-PlaY Reinforcement Learning method inspired by social deduction game Who Is the Spy?. Agents receive asymmetric information, complete the same target task, and vote to identify a designated spy. Because the spy identity is predetermined, voting outcomes provide fully verifiable rewards, while successful identification remains closely related to output quality. Experiments on text summarization, creative writing, and mathematical reasoning show that SpyRL outperforms existing self-improvement methods on non-verifiable tasks and yields consistent gains on verifiable reasoning tasks. These results demonstrate that task transformation can extend scalable RLVR-based self-improvement beyond inherently verifiable domains. Models and code have been released at https://github.com/wangqinsi1/RLSVR/tree/SpyRL.
Executive Summary: Executive Summary
Reinforcement Learning with Verifiable Rewards (RLVR) has driven recent gains in reasoning models by supplying scalable, unbiased training signals in domains such as mathematics and coding. Open-ended tasks like summarization and creative writing lack deterministic verifiers, so practitioners rely on human preferences, reward models, or LLM judges. These substitutes introduce evaluation bias, cap performance at the judge’s capability, and add repeated inference cost. The paper addresses this verifiability bottleneck by extending the self-supervised learning principle of task transformation to reinforcement learning.
The authors propose Reinforcement Learning with Self-Verifiable Rewards (RLSVR), a paradigm that converts an open-ended task into a proxy environment whose internal rules and interaction outcomes automatically generate deterministic rewards. They instantiate RLSVR with SpyRL, a multi-agent self-play framework modeled on the social-deduction game “Who Is the Spy?” In each round, most agents receive the full task input while one “spy” receives a degraded version; all agents complete the same target task and then vote to identify the spy. Because the spy’s identity is assigned by the environment, voting outcomes supply fully verifiable rewards. The design ensures that successful detection depends on output quality, so the reward signal remains tightly coupled to the original task objective.
Experiments across text summarization, creative writing, and mathematical reasoning show that SpyRL substantially outperforms prior self-play baselines on non-verifiable tasks and yields further gains on already-verifiable ones. On Qwen3-8B, SpyRL records 75.4 % and 77.3 % win rates against baselines on summarization and creative writing, respectively, while also improving mathematical reasoning accuracy by 8.97 % and 6.16 % on Qwen3-4B and 8B across seven benchmarks. Human evaluations and cross-judge tests confirm that outputs receiving fewer suspicion votes are rated higher in quality, novelty, and coherence. The method requires no external reward model or additional verifier cost and demonstrates positive cross-task transfer between summarization and creative writing.
These results indicate that verifiability need not be an intrinsic property of a task; it can be engineered through controlled information asymmetry and collective decision-making. SpyRL therefore removes a key obstacle to scalable self-improvement on open-ended capabilities. Senior leaders should consider pilot deployments of SpyRL-style training for summarization, writing, and other generation workloads where preference data are expensive or noisy. Immediate next steps include testing larger group sizes, additional task transformations, and longer training runs to assess stability at scale. The main limitations are the need for task-specific degradation operators and the current evaluation scope limited to three domains and two model families; further work should quantify generalization to new domains and measure any degradation risks associated with extended self-play.
1. Introduction
Section Summary: Reinforcement Learning with Verifiable Rewards works well for clear-cut tasks like math and coding but struggles with open-ended activities that involve subjective judgment. Drawing from self-supervised learning, where models create their own training signals from data without human labels, the authors introduce Reinforcement Learning with Self-Verifiable Rewards, a framework that converts such open-ended tasks into environments with automatically checkable rewards. They demonstrate this idea through SpyRL, an information-asymmetric self-play system modeled on a social deduction game, which produces measurable improvements in summarization, creative writing, and mathematical reasoning.
Reinforcement Learning with Verifiable Rewards (RLVR) has enabled scalable training for reasoning models like OpenAI o1 ([1]) and DeepSeek-R1 ([2]). However, while RLVR excels in deterministic domains like math and coding, it remains brittle in open-ended tasks requiring subjective judgment. To bridge this gap, prior research relaxes strict verifiability using learned preference signals (e.g., RLHF ([3]), DPO ([4])) or model-based feedback proxies, such as LLM-as-a-Judge ([5]) and self-rewarding mechanisms ([6]). These methods broaden the applicability of RL, but they also introduce evaluation bias, bottlenecks in judge capability, and additional inference costs.
Self-supervised learning offers a useful methodological precedent for addressing this challenge. In the absence of human annotations, it constructs pretext objectives whose supervisory signals are derived automatically from the data itself ([7, 8]). For example, masked language modeling learns contextual representations by recovering masked tokens ([9]), while contrastive learning captures semantic structure by distinguishing related views from unrelated samples ([10, 11]). Although these pretext objectives differ from the ultimate downstream tasks, they can induce transferable representations and capabilities ([8, 9, 11]). Their success suggests a broader principle: when a proxy objective can generate supervision automatically while preserving substantial capability overlap with the target task, learning can proceed without task-specific human annotation.
Motivated by the principle underlying self-supervised learning, we extend its task-transformation paradigm to RLVR and propose Reinforcement Learning with Self-Verifiable Rewards (RLSVR). RLSVR can be viewed as self-supervised learning for RLVR: it transforms an original open-ended task into a proxy environment in which rewards can be automatically verifiable, allowing the reward signal to arise from the transformed task environment itself. Here, self-verifiable means that reward verifiability is derived from the environment’s internal rules or interaction outcomes. Through this transformation, tasks that originally lack a verifier can obtain scalable, verifiable rewards.
Building on RLSVR, we introduce a concrete instantiation, SpyRL (Self-PlaY Reinforcement Learning), an information-asymmetric self-play framework that transforms open-ended generation tasks into a multi-agent environment resembling the social deduction game Who Is the Spy?. In each round, multiple agents receive asymmetric information: most civilian agents are given the complete task input, whereas a single spy agent receives only a degraded version. All agents must then perform the same target task based on their respective observations, such as producing a summary, writing a story, or constructing and solving a mathematical problem. The agents subsequently inspect one another’s outputs and vote on which participant is the spy. Because the spy’s identity is predetermined by the environment, the correctness of the vote is fully verifiable. At the same time, whether an agent is suspected is closely tied to whether its output reveals deficiencies in the information available to it. In this way, SpyRL converts output quality, which is otherwise difficult to evaluate directly, into a verifiable training signal induced by information asymmetry and identity inference.
We evaluate the effectiveness of SpyRL on text summarization, creative writing, and mathematical reasoning tasks. Experimental results show that SpyRL significantly outperforms existing self-improvement methods on two representative non-verifiable tasks, while also yielding consistent gains on verifiable mathematical reasoning tasks. Further analysis reveals that outputs receiving more suspicion votes tend to be of lower quality, indicating a strong correlation between the rule-based rewards induced by identity inference and actual task performance. These findings support our central claim: with appropriate task and environment transformations, non-verifiable open-ended tasks can be incorporated into a scalable RLVR training paradigm.
Experimental results show that our method consistently outperforms existing self-evolution approaches ([12, 13]). On Qwen3-8B, SpyRL achieves $75.4%$ and $77.3%$ win rates on summarization and creative writing, significantly outperforming existing self-play methods that yield only marginal gains. SpyRL also leads in verifiable tasks, improving Qwen3-4B and 8B by $8.97%$ and $6.16%$ in mathematical reasoning across seven benchmarks. Overall, the contributions of this paper are as follows:
- RLSVR: A Self-Supervised Learning Paradigm for RLVR: We propose Reinforcement Learning with Self-Verifiable Rewards (RLSVR), which brings the central idea of self-supervised learning—generating supervision through pretext tasks—into RLVR. By transforming tasks and environments, RLSVR constructs self-verifiable rewards for open-ended tasks that lack directly verifiable feedback.
- SpyRL: An RLSVR Instantiation Based on Information-Asymmetric Self-Play: We introduce SpyRL, which reformulates open-ended tasks as a multi-agent social-deduction game resembling Who Is the Spy?. Environment-assigned hidden identities and rule-based voting outcomes automatically produce verifiable rewards, enabling scalable model self-improvement.
- Validation Across Diverse Tasks: We evaluate SpyRL on text summarization, creative writing, and mathematical reasoning. The results show that SpyRL substantially improves performance on non-verifiable open-ended tasks while also providing consistent gains on reasoning tasks with existing verifiable rewards.
- Implications for Open-Ended Self-Improvement: Our findings suggest that self-play need not be restricted to inherently verifiable domains such as mathematics and code. Through task transformation and environment-induced rewards, self-play has the potential to become a scalable paradigm for training open-ended capabilities without human supervision.
2. RLSVR: Reinforcement Learning with Self-Verifiable Rewards
Section Summary: RLSVR tackles the core limitation of standard reinforcement learning with verifiable rewards, where automatic checkers work only for tasks with clear right answers but fail for open-ended work such as writing or summarization. It converts the original task into a proxy setting by injecting a hidden latent variable, having the model perform its usual work under conditions tied to that variable, and then scoring the outputs through simple rule-based questions about the hidden variable that can be answered correctly only if the work was high quality. The resulting rewards are generated automatically from the data itself, allowing the same efficient training methods used for math or coding problems to be applied to previously unverifiable tasks.
The verifiability bottleneck of RLVR. We consider a target task defined by an input distribution $\mathcal{D}$ and a task instruction $\tau$ (e.g., "summarize the following report"). Given an input $x \sim \mathcal{D}$, a policy $\pi_\theta$ generates an output $y \sim \pi_\theta(\cdot \mid x, \tau)$. RLVR optimizes
$ \max_\theta ; \mathbb{E}{x \sim \mathcal{D}, ; y \sim \pi\theta(\cdot \mid x, \tau)} \big[V(x, y) \big],\tag{4} $
where $V(x, y) \in {0, 1}$ is a deterministic verifier such as an answer checker or a unit-test executor. RLVR scales precisely because $V$ provides unbiased, unlimited, and essentially free supervision. For open-ended tasks such as creative writing or summarization, however, the true objective is a latent quality function $Q(x, y)$ for which no verifier exists. Prior work replaces $V$ with an approximate evaluator $\hat{V}$ —a learned reward model, an LLM judge, or rubric-based scoring—but this reintroduces evaluation bias, caps the policy at the evaluator's competence, and adds inference cost for every rollout. The root cause is that these methods try to approximate the unverifiable objective $Q$ directly.
Task transformation. Self-supervised learning faces an analogous dilemma and resolves it differently: instead of approximating missing labels, it transforms the task into a pretext objective (e.g., masked token recovery) whose labels are generated automatically from the data itself. RLSVR extends this idea from labels to rewards. Concretely, a task transformation $\Phi$ maps the original task $(\mathcal{D}, \tau)$ to a proxy environment $\mathcal{E}$ that operates as follows:
- Latent-variable injection. The environment samples an input $x \sim \mathcal{D}$ and a latent variable $z$, and records $z$ as the episode's ground truth. The latent variable can take many forms: which of several inputs has been perturbed, which portion of $x$ has been withheld, under which condition each output will be generated, and so on. Its realization is never directly revealed to the policy.
- Conditioned task execution. The environment constructs one or more observations $o$ from $(x, z)$ and the policy performs the original target task on each observation, producing $y \sim \pi_\theta(\cdot \mid o, \tau)$. This step ensures that the capabilities exercised in $\mathcal{E}$ remain those of the target task.
- Verifiable interaction. The environment's rules then pose a question about $z$ that must be answered from the task outputs alone—e.g., identifying which output was produced under the hidden condition, or recovering the withheld information. Crucially, the transformation is designed so that answering this question correctly hinges on the quality of the outputs from step 2.
- Rule-based reward. The environment computes the reward $R$ by checking the interaction outcomes against the recorded $z$.
We call the resulting reward self-verifiable: it is a deterministic, rule-based function of the environment-assigned $z$ and the observable interaction outcomes, and thus requires no human annotation, no learned reward model, and no external judge. The key property is that ground truth exists by construction—since $z$ is sampled by the environment itself, any prediction about $z$ can be checked exactly, just as a math verifier checks a final answer. In this sense, RLSVR is self-supervised learning for RLVR: the transformation $\Phi$ plays the role of the pretext task, the latent variable $z$ plays the role of the automatically generated label, and standard RLVR machinery (e.g., GRPO) applies directly to $\mathcal{E}$, with $R$ replacing the unverifiable $Q$ in Equation 4.
3. SpyRL: Instantiating RLSVR via Information-Asymmetric Self-Play
Section Summary: SpyRL turns hard-to-verify tasks such as writing or summarization into a two-stage game in which most players receive full task information while one spy receives a degraded version. All players produce outputs, then collectively vote to identify the spy; because the spy’s identity is known to the system, the votes supply automatic, rule-based rewards that reward strong performance and accurate detection. This closed loop replaces external judges with information asymmetry and peer competition, letting models improve at open-ended work through self-play.
In this section, we present SpyRL, a multi-agent self-play framework for tasks where direct reward verification is infeasible. SpyRL couples a performing stage and a detection stage into a closed-loop game, transforming the unverifiable objective of output quality (i.e., $Q$ in Section 2) into a verifiable identity-recognition problem—thereby producing rule-based training signals without any external verifier.
3.1 Overview
As illustrated in Figure 2 and summarized in Algorithm 1, each SpyRL training epoch alternates between Performing and Detection stages. During the Performing stage, $n-1$ civilians with full information and one spy with corrupted information generate outputs for the target task. In the Detection stage, players jointly analyze these outputs to identify the spy. Because the environment explicitly assigns this identity, the detection yields a naturally verifiable outcome. Crucially, the rewards across both stages are tightly coupled: the preassigned identity supervises the detection stage, whose outcomes simultaneously determine the performing stage's reward. This mutual optimization transforms the unverifiable objective of output quality into a computable surrogate: whether an output reveals underlying information deficiency.
This design yields two key advantages. First, by extracting stable training signals from identity recognition, SpyRL eliminates reliance on external verifiers, naturally extending RL to open-ended domains like creative writing. Second, unlike pointwise evaluations relying on single proposer–solver pairs ([12, 13]), SpyRL leverages multi-player competition and collective decision-making. This mitigates bias amplification from isolated verifiers, establishing a robust self-play mechanism for non-verifiable tasks.
3.2 Information-Asymmetric Performing Stage
Capability-Oriented Task Design. In this stage, players execute the target task based on their allocated information, generating outputs for subsequent identity judgment. To genuinely enhance model proficiency, the capabilities required in the performing stage must closely align with the target task. Following this, we construct three representative tasks: mathematical reasoning, creative writing, and text summarization (detailed in Figure 3). Notably, the performing stage relies solely on cheap document-level information, eliminating the need for expensive, manually designed question-level supervision. This allows SpyRL to efficiently transfer across diverse scenarios.
Asymmetric Information Allocation. To introduce verifiable training signal into the performing stage, SpyRL assigns asymmetric information inputs to different players. Formally, in each epoch, we first sample an instance $x \sim \mathcal{D}$ from the task distribution, and then uniformly sample an spy index $u \sim \mathrm{Unif}({1, \dots, n})$. We then construct the private observation $o_i$ for each player $i$ as
$ o_i = \begin{cases} x, & i \neq u, \ g(x), & i = u, \end{cases}\tag{5} $
where $g(\cdot)$ denotes an information-degradation operator applying controlled information loss to the original material, inducing a task-relevant disadvantage for the spy. The operator $g$ can take various forms, such as context truncation or key information compression. Importantly, $g$ obscures only the critical information necessary for task completion while preserving style, length, and thematic consistency. This prevents detectors from exploiting superficial shortcuts. Figure 3 details this mechanism across tasks. Ultimately, this asymmetric allocation ensures the spy exhibits inferior performance, establishing a naturally verifiable reward signal for the subsequent detection stage.
Given the observation $o_i$, each player generates an output $y_i$ based on the policy $\pi^P_\theta$:
$ y_i \sim \pi^P_\theta(\cdot \mid o_i, \tau)\tag{6} $
where $\tau$ represents the unified instruction template for the target task. All outputs generated are subsequently revealed in the detection phase and serve as the basis for identity inference. Consequently, to avoid being voted out as the spy, players are incentivized to perform at their highest capacity on the target task—e.g., providing clearer and more rigorous derivations in mathematical reasoning, or generating more natural and innovative content in creative writing. Most importantly, this requirement for "better performance" is defined by relative superiority over peer players within the same group, driving the continuous self-evolution of the players throughout the training process.
3.3 Detection Stage with Verifiable Rewards
Verifiable Identity Detection.
In the detection stage, each player is required to infer the identity of the spy player and cast a vote based on the outputs in the performing stage. Let the set of public outputs be $Y = {y_1, \dots, y_n}$. Then, the detection state of player $i$ in this stage is defined as $s_i = (o_i, Y).$ Based on this state, the player samples a voting action according to the policy $\pi^D_\phi$:
$ v_i \sim \pi^D_\phi(\cdot \mid s_i), \qquad v_i \in {1, \dots, n},\tag{7} $
where $v_i = j$ means that player $i$ believes player $j$ is the spy. The pivotal advantage of this stage lies in its inherently verifiable reward signal. Since the spy identity $u$ is explicitly specified by the environment, whether a player's detection is correct can be computed directly. Accordingly, the base reward for detector $i$ is defined as $r_i^D = \mathbb{I}[v_i = u]$, where $\mathbb{I}[\cdot]$ denotes the indicator function.
Group-based Advantage.
To obtain an optimization signal with lower variance that is better suited for collective decision-making, we adopt a GRPO-style ([14]) group relative advantage formulation. For the $n$ detector rewards within the same epoch, let their mean and standard deviation be denoted as $\mu_D$ and $\sigma_D$, respectively. The normalized advantage for the $i$-th detector is then computed as,
$ \mu_D = \frac{1}{n}\sum_{i=1}^n r_i^D, \qquad \sigma_D = \sqrt{\frac{1}{n}\sum_{i=1}^n \left(r_i^D-\mu_D\right)^2}, \qquad A_i^D = \frac{r_i^D - \mu_D}{\sigma_D + \epsilon}.\tag{8} $
This within-group normalization based on relative performance allows the detector to learn without relying on an additional critic network; instead, its optimization signal is determined directly by relative performance among players in the same group. Unlike training paradigms that rely on a single verifier, the detection process in SpyRL is inherently collective: a misjudgment by a single detector does not dictate the overall optimization trajectory; rather, it is counterbalanced by the voting outcomes of other players. Therefore, the detection stage not only provides a stable and verifiable reward signal, but also, through group-based aggregation, serves as a more robust source of supervision for the performing stage that is less vulnerable to local biases.
Input: task distribution $\mathcal{D}$, player number $n$, epochs $T$, instruction $\tau$, degradation operator $g(\cdot)$, performer policy $\pi_\theta^P$, detector policy $\pi_\phi^D$, coefficients $\beta,\lambda$
Output: trained $\pi_\theta^P,\pi_\phi^D$
for $t \leftarrow 1$ to $T$:
beginalgoinferbox
Information Allocation: Assign asymmetric observations
Sample $x \sim \mathcal{D}$ and spy index $u \sim \mathrm{Unif}(\{1,\dots,n\})$
for $i \leftarrow 1$ to $n$:
$o_i \leftarrow x$ if $i \neq u$, else $o_i \leftarrow g(x)$
// Observations are cheap document information.
Performing Stage: Perform on the target task under private observations.
for $i \leftarrow 1$ to $n$:
$y_i \sim \pi_\theta^P(\cdot \mid o_i,\tau)$
// Performing tasks are the same/related to target task.
Detection Stage: Infer spy identity from public outputs $Y \leftarrow \{y_i\}_{i=1}^n$.
for $i \leftarrow 1$ to $n$:
$v_i \sim \pi_\phi^D(\cdot \mid (o_i, Y)), \quad v_i \in \{1,\dots,n\}$
// Inference based on whose output is the worst.
endalgoinferbox
beginalgoupdatebox
Update Phase
Detection Reward: Determined by whether detected the correct spy.
for $i \leftarrow 1$ to $n$:
$r_i^D \leftarrow \mathbb{I}[v_i = u]$
// verifiable rewards from environment-assigned identity.
$\mu_D \leftarrow \frac{1}n\sum_{i=1}^n r_i^D,\quad \sigma_D \leftarrow \sqrt{\frac{1}n\sum_{i=1}^n (r_i^D-\mu_D)^2},\quad A_i^D \leftarrow \frac{r_i^D-\mu_D}{\sigma_D+\epsilon}$
Performing Reward: Determined by the number of votes received.
$m_j \leftarrow \sum_{i=1}^n \mathbb{I}[v_i=j], \quad \forall j\in\{1,\dots,n\}$ // non-verifiable rewards made by detectors.
$r_u^P \leftarrow -\beta(m_u-\bar{m}_c)$
$r_j^P \leftarrow \frac{\beta}{|\mathcal{C}|}(m_u-\bar{m}_c)-\lambda(m_j-\bar{m}_c), \quad \forall j \in \mathcal{C}$, $\mathcal{C} \leftarrow \{j:j\neq u\}$
// Self-play between civilian \& spy.
$A_i^P \leftarrow$ Role Advantage Estimation $(r_i^P,\mathrm{role}_i)$
endalgoupdatebox
Update performer policy $\pi_\theta^P$ using Eq. 10;
Update detector policy $\pi_\phi^D$ using Eq. 11
3.4 Two-Stage Coupled Optimization
The training of SpyRL lies in the two-stage coupled optimization: The voting results from the detection stage define the reward for the performing stage, while the quality of the performers' outputs in turn determines the difficulty of detection. The two stages are therefore mutually dependent and shaping, forming a closed-loop learning system.
Zero-Sum Reward for Performers. The reward design in the performing stage follows two principles. First, a player who is suspected by more peers should receive a lower reward. Second, the total reward between the spy and the civilian players should remain zero-sum. Driven by this objective, we define the rewards for the spy player $u$ and the civilian players $c_j$ in the performing stage as
$ r_u^P = -\beta\left(m_u - \bar{m}c\right), \qquad r{c_j}^P = \frac{\beta}{n_c}\left(m_u - \bar{m}c\right) - \lambda\left(m{c_j} - \bar{m}_c\right), \qquad j = 1, \dots, n_c,\tag{9} $
where $m_u$ denotes the number of votes received by the spy player, $m_{c_j}$ denotes the number of votes received by the $j$-th civilian player, and $\bar{m}c$ denotes the average number of votes received by all civilian players. Here, $\beta > 0$ controls the strength of the competitive signal between the spy and civilian players, and $\lambda > 0$ adjusts the intra-group consistency penalty among civilian players. This reward design naturally satisfies desirable properties. (1) Zero-sum Constraint: $r_u^P + \sum{j=1}^{n_c} r_{c_j}^P = 0$, which enables continual co-evolution between the spy and civilian players through competition. (2) Within-group Competition: If a particular civilian player receives substantially more votes than the others, it incurs a larger penalty. This ensures that the learning signal is fundamentally relative: instead of directly optimizing an unverifiable "task quality" score, the model learns to produce outputs that are better than those of the other players under the same information setting.
Due to the structural information asymmetry between the spy and the civilians in the performing stage, their raw reward distributions are typically unbalanced. Directly employing the aforementioned returns for policy optimization is prone to inducing systemic bias in the advantage estimation across different roles. To alleviate this issue, we incorporate Role-Advantage Estimation (RAE) ([15]) during the optimization of the performing stage to explicitly calibrate the role biases induced by information asymmetry. Detailed formulations of RAE are provided in Appendix C.1.
Iterative Optimization. During optimization, we update the performer policy $\pi_\theta^P$ and the detector policy $\pi_\phi^D$ separately. We adopt a GRPO-style clipped objective together with a KL regularization term against a reference policy to suppress policy drift.
For the performing stage, let $\rho_{k, t}^P$ denote the probability ratio between the updated and old performer policies at token $t$ of sample $k$. The optimization objective is defined as
$ \mathcal{L}P(\theta) = -\mathbb{E}\Bigg[\frac{1}{n}\sum{k\in{u}\cup\mathcal{C}}\sum_t \min\Big(\rho_{k, t}^P A_k^P, , \mathrm{clip}(\rho_{k, t}^P, 1-\epsilon, 1+\epsilon)A_k^P \Big) \Bigg] + \beta_P, \mathrm{KL}(\pi_\theta^P\Vert \pi_{\mathrm{ref}}^P),\tag{10} $
where $A_k^P$ is constructed from the zero-sum reward in the performing stage together with the role-advantage estimation described above.
For the detection stage, we use the group-based advantage $A_i^D$ defined in Section 3.3 to optimize the detector policy. Let $\rho_i^D$ denote the action-level probability ratio between the updated and old detector policies. The detection-stage objective is
$ \mathcal{L}D(\phi) = -\mathbb{E}\Bigg[\frac{1}{n}\sum{i=1}^{n} \min\Big(\rho_i^D A_i^D, , \mathrm{clip}(\rho_i^D, 1-\epsilon, 1+\epsilon)A_i^D \Big) \Bigg] + \beta_D, \mathrm{KL}(\pi_\phi^D\Vert \pi_{\mathrm{ref}}^D).\tag{11} $
During training, we alternate between optimizing the performing stage and the detection stage to avoid premature convergence to a local equilibrium. Intuitively, once the detectors can identify the spy player relatively easily, further strengthening the detection stage often yields only limited gains. It is more effective to shift optimization to the performers, encouraging them to generate outputs that are both higher-quality and more deceptive. Conversely, when the behavior distribution induced by the performing stage substantially increases the difficulty of identification and leads to a drop in detector performance, training switches back to the detection stage to restore its discriminative ability. In this manner, the two stages continuously engage in a dynamic interplay revolving around the model's current capability frontier, rather than stagnating after over-optimizing either side.
This alternating optimization strategy offers two primary benefits. First, it explicitly breaks the policy stagnation commonly observed in fixed self-play ([16]), allowing performers and detectors to continually shape each other and thereby maintain stable learning pressure. Second, compared with updating both policies simultaneously throughout training, stage-wise alternation reduces interference in credit assignment, so that each round of updates focuses on the current dominant bottleneck. As a result, it typically leads to better training stability and higher sample efficiency. We provide additional implementation details of the alternating optimization strategy in Appendix C.2.
4. Experiments
Section Summary: The experiments test SpyRL on three tasks—summarizing government reports, writing creative stories, and solving math problems—using large-scale training that alternates between players generating responses and trying to spot the single “spy” who received incomplete input. Models are trained with groups of five responses per example and compared against prior self-play methods using automatic metrics, GPT-4o judgments, and human ratings. Results show the method improves performance across both tasks with clear answers and those without, while the number of suspicion votes received reliably tracks lower-quality outputs.
4.1 Experimental Setting
Environment Setup. We evaluate our method across three domains that cover substantially different generation and reasoning capabilities. In every domain, civilian players receive the complete, unmodified input, whereas the spy receives a corrupted version in which one continuous span is masked. The masked span is sampled according to a domain-specific masking ratio, requiring the spy to infer the missing information from the remaining context before producing its response. All players are then asked to generate outputs appropriate for the corresponding task. (1) Text Summarization uses GovReport, a dataset of long government reports that require comprehensive and factually grounded summaries. We mask $20%$ of the input and instruct each player to generate a summary that captures the main arguments, findings, and supporting details. (2) Creative Writing uses WritingPrompts, where players produce open-ended stories conditioned on a natural-language prompt. We again mask $20%$ of the input, creating uncertainty about part of the original narrative premise while preserving sufficient context for coherent generation. (3) Mathematical Reasoning uses Nemotron-CC-Math-v1. In this setting, $40%$ of the source text is masked, and players must formulate and solve a mathematical question based on the information available in the input. The higher masking ratio reflects the greater redundancy and structural regularity of mathematical text.
Training Setup. We train all models with a batch size of 1024 for 100 epochs and set the maximum generation length to 2048 tokens. Unless otherwise specified, each training instance uses a group size of $n=5$, meaning that five candidate responses are sampled for the relevant group-based training objective. We study the effect of this group size separately in Section 4.4. Training strictly alternates between the performing and detection stages: one stage is updated while the other remains fixed, after which their roles are exchanged. This alternating schedule prevents the policy and its evaluator from changing simultaneously and helps maintain a stable learning signal throughout training. We use the same general training protocol across all three domains, while adapting task-specific prompting and evaluation procedures where necessary. A complete list of optimization settings, decoding parameters, and other implementation details is provided in Appendix C.3.
Baselines & Metrics. We compare our approach with two state-of-the-art proposer-solver self-play frameworks, R-Zero ([13]) and Absolute Zero ([12]). These baselines provide strong reference points for evaluating whether our method improves self-play training beyond existing approaches. Performance is measured using task-specific automatic metrics, GPT-based pairwise evaluation and human evaluation. The automatic metrics capture domain-dependent properties such as summary quality, generation quality, and mathematical correctness. For the pairwise evaluation, GPT-4o performs A/B comparisons between outputs produced by the trained model and those produced by the corresponding base model. To mitigate position bias, we evaluate every pair in both possible presentation orders and aggregate the resulting judgments. This procedure ensures that the reported preference rates are not driven by whether a response appears first or second. Additional information about baseline implementation, evaluation prompts, metric definitions, and aggregation procedures is included in Appendix C.4.
::: {caption="Table 1: Results on summarization benchmarks. We report the ROUGE-L of every method together with the GPT-4o pairwise A/B win rate (%) of SpyRL against the method in that row. Win rates of each method against its own untrained backbone are reported in Appendix D.1."}
:::
4.2 Main Results
SpyRL is universally compatible with both verifiable and unverifiable tasks. We present the performance of models trained on the three distinct categories of tasks in Table 1–Table 3.
From Non-verifiable Reward to Verifiable Reward. To demonstrate that SpyRL can reliably transform non-verifiable rewards into verifiable training signals, we conduct an additional validation experiment. As shown in Figure 4, we run 100 games for both summarization and creative writing, and record the number of votes each player receives in the detection stage, i.e., the number of times the player is suspected of being the spy. At the same time, for each game, we ask GPT-4o to rank the outputs of the five players in the performing stage by quality from best to worst ($1=\text{best}$, $5=\text{worst}$), and collect the rank of each player. We then compute the average GPT-4o rank for players receiving different numbers of votes. As illustrated in Figure 4, the number of votes a player receives is positively correlated with its rank number. In other words, lower-quality outputs tend to attract more suspicion votes and thus receive smaller rewards during training; conversely, higher-quality outputs tend to receive fewer votes and obtain larger rewards. These results show that, in SpyRL, the reward assigned in the performing stage is directly aligned with task performance, enabling continuous performance improvement throughout training. Importantly, this improvement does not rely on any external verifier. Instead, SpyRL achieves it by cleverly converting a non-verifiable quality objective into a verifiable identity-discrimination problem.
::: {caption="Table 2: Results on creative writing benchmarks. We report GPT-4o pairwise win rates (%) of SpyRL against the untrained backbone and against each self-evolution baseline. A/B win rates of the baselines against their own backbone are reported in Appendix D.1."}
:::
::: {caption="Table 3: Results on mathematical and general reasoning benchmarks. We report accuracy (%) on five mathematical benchmarks and two broader reasoning benchmarks."}
:::
::: {caption="Table 4: Human evaluation results on creative writing. We report the pairwise win rates of SpyRL against each baseline across five evaluation dimensions. Higher is better."}
:::
Performance on Non-verifiable Tasks. The results on non-verifiable tasks are presented in Table 1 and Table 2. Overall, SpyRL achieves the best performance on both summarization and creative writing, two representative open-ended generation tasks. On summarization, SpyRL attains the highest ROUGE-L on every benchmark and for both backbones, improving over Absolute Zero on GovReport from 33.2 to 36.7 with Qwen3-4B and from 32.5 to 34.1 with Qwen3-8B. The ABTest result show that SpyRL wins the majority of comparisons in all thirty cells. The same pattern holds on creative writing (Table 2), where SpyRL is preferred in every fine-grained dimension against the backbone and both baselines. The margins are largest in novelty and emotion, indicating that the gains are not confined to surface fluency or structural regularity but also extend to more subjective aspects of open-ended generation. The two baselines, in contrast, gain far less from their own self-evolution procedure. This is because these methods rely heavily on task verifiability: they depend on verifiable solver feedback to dynamically adjust difficulty. SpyRL, however, reformulates the quality objective as an identity-recognition problem, thereby encouraging the model to produce outputs that are consistently stronger and more convincing than those of other players. This design allows SpyRL to deliver significant gains on open-ended tasks.
Performance on verifiable Tasks. The results on verifiable tasks are shown in Table 3. Overall, SpyRL again achieves the best performance on mathematical and verifiable reasoning tasks. For mathematics, SpyRL achieves the best performance across all five benchmarks for both Qwen3-4B and Qwen3-8B. The gains are particularly pronounced on more challenging benchmarks such as AIME24 and AIME25. Meanwhile, SpyRL also achieves the best results on broader reasoning benchmarks such as MMLU-Pro and GPQA-D, suggesting that its advantage is not limited to pure mathematical problem solving, but generalizes to a wider range of verifiable reasoning scenarios. Compared with its performance on non-verifiable tasks, R-Zero and Absolute Zero indeed show more noticeable gains on verifiable tasks. Nevertheless, SpyRL still consistently outperforms these methods even in this setting. This suggests that the advantage of SpyRL does not merely stem from task verifiability itself. Rather, through reward transformation and within-group competition, SpyRL provides a finer-grained and more stable learning signal, encouraging the model to generate reasoning processes that are more rigorous, complete, and persuasive. In addition, compared with the conventional single proposer-solver paradigm ([12, 13]), the group-based design of SpyRL mitigates the bias introduced by single-sample optimization, which further contributes to its strong gains on verifiable tasks. We also report performance improvements on harder tasks in Appendix D.2 (Table 15).
::: {caption="Table 5: A/B Test Comparison with rubric-as-reward baselines. We report the pairwise win rates of SpyRL against Qwen3.5-27B-RaR and GPT-4o-RaR. SpyRL uses no external verifier, whereas Qwen3.5-27B-RaR and GPT-4o-RaR incur approximately 200 and 900 in additional verifier costs, respectively, in our experiments."}
:::
::: {caption="Table 6: Results on scientific and domain-specific summarization benchmarks. SpyRL is trained on the PubMed corpus with the same game construction and hyperparameters as the main summarization run, then evaluated on arXiv, PubMed, and BillSum. We report ROUGE-L (R-L) and GPT-4o A/B test win rates (%) against the untrained base model. The shaded row is SpyRL; the Average columns are means over the three benchmarks."}
:::
Human Evaluation. We further conduct a blinded human evaluation on creative writing to assess whether the gains of SpyRL align with human preferences. Ten Ph.D. students evaluate 400 randomly sampled prompts, with 200 from WritingPrompts and 200 from WritingBench; each evaluator assesses 40 instances. For each prompt, evaluators rank four anonymized responses from Qwen3-4B, R-Zero, Absolute Zero, and SpyRL across five dimensions. We report the pairwise win rate of SpyRL whenever its response is ranked above the corresponding baseline. As shown in Table 4, SpyRL achieves overall win rates of 80.0%, 78.5%, and 74.0% against Qwen3-4B, R-Zero, and Absolute Zero on WritingPrompts, respectively, with similarly strong results on WritingBench. SpyRL also consistently leads across all fine-grained dimensions, particularly in novelty and emotion. These results confirm that the improvements of SpyRL are recognized by human evaluators. We also report the agreement between human and LLM evaluations in Appendix D.5 (Table 19).
Comparison with Rubric-as-Reward Baselines. We further compare SpyRL with rubric-as-reward (RaR) methods that rely on external evaluators. Using the same GRPO framework, we train for 50 iterations with a batch size of 1024, employing Qwen3.5-27B and GPT-4o as rubric executors to construct Qwen3.5-27B-RaR and GPT-4o-RaR. As shown in Table 5, SpyRL outperforms Qwen3.5-27B-RaR across all dimensions on both WritingPrompts and WritingBench, with overall win rates of 59.3% and 56.2%. Against the stronger GPT-4o-RaR, SpyRL remains competitive, while performing better in novelty and emotion. Moreover, SpyRL requires no external verifier, whereas Qwen3.5-27B-RaR and GPT-4o-RaR incur approximately $200 and $900 in additional verifier costs, respectively. These results highlight the cost-performance trade-off of SpyRL and the benefit of jointly improving performing and detection capabilities through self-play, without being constrained by fixed verifier.
::: {caption="Table 7: Cross-task transfer results. Each row takes the SpyRL model trained on one task and evaluates it, without any further fine-tuning, on a task it was not trained on. We report the GPT-4o A/B win rate (%) against the corresponding untrained Qwen3-4B, so a value above 50% indicates positive transfer (↑) and a value below 50% negative transfer (↓). Shaded rows are the positive-transfer cases."}
:::
::: {caption="Table 8: Ablation study on Math500 accuracy (%) across training epochs. 'Only Performing' freezes the detection stage; 'Only Detection' freezes the performing stage; 'Without spy' removes information asymmetry. SpyRL with full two-stage coupled optimization."}
:::
4.3 Generalization Beyond the Main Benchmarks
Domain-Specific Summarization. The summarization experiments above all draw their training corpus from government reports, which leaves open whether SpyRL depends on that particular document distribution. We therefore repeat the summarization setting with scientific articles, training Qwen3-4B on the PubMed summarization corpus while keeping the game construction, prompts, and optimization settings unchanged: civilians receive the complete article, the spy receives the article with a continuous $20%$ span masked, and the detector identifies whose summary was most likely written from incomplete source material. Evaluated on arXiv, PubMed, and BillSum (Table 6), SpyRL raises ROUGE-L from 28.1 to 32.5, from 30.3 to 35.1, and from 41.3 to 46.8, an average gain of 4.9 points, with A/B win rates of 72.1%, 68.9%, and 69.5%. The benefit of information-asymmetric self-play therefore carries over to scientific and other domain-specific sources.
Cross-Task Transfer. We also ask whether the capabilities acquired in one domain carry over to another. We take each final checkpoint, evaluate it on a task it was not trained on, and compare it against the corresponding untrained Qwen3-4B under the same GPT-4o A/B protocol. As shown in Table 7, summarization and creative writing transfer positively in both directions. This is expected, since both tasks draw on shared capabilities such as content organization, completeness, discourse coherence, and long-range consistency. The mathematical-reasoning model, by contrast, falls below parity on both writing tasks; it primarily strengthens symbolic manipulation and multi-step reasoning, which overlap far less with the stylistic and discourse-level demands of open-ended writing.
::: {caption="Table 9: Ablation on Role-Advantage Estimation. Accuracy (%) of Qwen3-4B trained with and without RAE, using identical data, reward function, and optimization settings; the w/o RAE variant uses the raw performing rewards without subtracting the role-specific baseline."}
:::
::: {caption="Table 10: Sensitivity to the information-degradation operator g(·). ROUGE-L and GPT-4o A/B win rates (%) against the untrained base model for SpyRL trained with a continuous span-masking ratio of 20% and 40%, with all other settings unchanged."}
:::
4.4 Ablation Study
In this section, we conduct ablation studies to systematically disentangle the contributions of individual SpyRL components, alongside key hyperparameter analysis.
Ablation on Modules. Table 8 demonstrates that alternating optimization between the performing and detection stages is crucial. Training exclusively on the performing stage initially exhibits noticeable improvements by directly providing task-correlated rewards. However, as the model's intrinsic capabilities grow, maintaining a static detection module inevitably leads to reward distortion. The frozen detector struggles to accurately discern quality nuances among increasingly sophisticated responses, causing the training process to rapidly plateau and oscillate. Conversely, training solely on the detection stage yields negligible target-task improvements. Furthermore, removing the spy mechanism yields a similar bottleneck phenomenon. Although the model achieves initial gains, its overall performance quickly stagnates. This underscores that without the auxiliary supervision and adversarial signals provisioned by the spy, the detection module cannot continuously co-evolve. Consequently, it fails to supply sufficiently accurate rewards to the performing stage. We also report an ablation study of alternating training in Appendix D.3, Table 16.
Ablation on Group Size. As shown in Figure 5, as the number of players ($n$) increases, the performance of SpyRL improves consistently across tasks. This suggests that enlarging the group size elevates the complexity and difficulty of the adversarial environment, thereby generating more rigorous and effective training signals that augment model performance. Notably, scaling the player count from 3 to 5 yields the most pronounced marginal gain, with the average performance surging from 5.5 to 9.3. As the player count further expands to 6 and 8, the performance improvements plateau, exhibiting a trend of diminishing marginal returns. This indicates that a group size of 5 already provides a sufficiently complex game environment to drive robust self-improvement.
Ablation on Role-Advantage Estimation. Because the spy and the civilians face structurally different task difficulty, their raw performing rewards are not directly comparable, which is what RAE is designed to correct. To verify that this calibration matters, we train a variant that is identical in data, reward function, and optimization settings, except that the raw rewards are used directly without subtracting the role-specific baseline. As shown in Table 9, removing RAE lowers the seven-benchmark average from 50.4 to 37.5, and degrades GSM8K, Math500, Minerva, MMLU-Pro, and GPQA-Diamond relative to the backbone. Without role-aware calibration, the optimizer conflates the spy's information disadvantage with poor policy quality, so the resulting gradient actively harms the model rather than merely slowing it down.
Sensitivity to the Degradation Operator. Since $g(\cdot)$ is the only component of SpyRL that must be specified per task, we ablate how much its design matters by comparing continuous span-masking ratios of $20%$ and $40%$ for summarization while holding everything else fixed. As shown in Table 10, the two settings are nearly indistinguishable; neither ratio is uniformly better across the five benchmarks. This insensitivity is expected, because continuous masking removes task-relevant content while preserving topic and surface form, and RAE re-centers rewards on each role's own baseline, so a player receives a positive signal whenever it beats its role-specific expectation regardless of how hard the operator makes the game overall. In practice, $g(\cdot)$ therefore requires little task-specific engineering, provided that the asymmetry it induces is meaningful but non-degenerate.
5. Conclusion
Section Summary: The paper introduces RLSVR, a new approach that adapts reinforcement learning for open-ended tasks by redesigning them into self-play games where rewards can be checked automatically, rather than relying on outside judges or models. In their SpyRL example, a hidden spy identity turns judging output quality into a simple recognition problem, leading to strong results on summarization, creative writing, and math tasks that beat prior self-improvement methods. Overall, the work shows that verifiability can be built into tasks through clever transformation, pointing toward scalable AI improvement without external reward systems.
We propose RLSVR, a training paradigm that extends RLVR to open-ended tasks by bringing the task-transformation principle of self-supervised learning into reinforcement learning: instead of approximating an unverifiable quality objective with external judges or reward models, RLSVR transforms the task into a proxy environment whose environment-assigned latent variables make rewards verifiable by construction. We instantiate this paradigm with SpyRL, an information-asymmetric self-play game in which a predetermined spy identity turns output-quality assessment into a verifiable identity-recognition problem. Across summarization, creative writing, and mathematical reasoning, SpyRL consistently outperforms existing self-improvement methods on open-ended tasks and yields further gains even on already-verifiable ones, with vote-based rewards shown to align closely with human and LLM quality judgments. Beyond the specific game, our results suggest a broader takeaway: verifiability need not be an intrinsic property of a task, but can be engineered through task transformation—opening a path toward scalable, verifier-free self-improvement on general open-ended capabilities.
Appendix
Section Summary: The appendix surveys prior research on multi-agent debate, self-play techniques, and reinforcement learning methods for language models. It describes how earlier work used games, debates, and self-generated tasks to boost reasoning and factuality, often in math or code domains, yet notes their reliance on verifiable answers, inference-time tricks, or indirect signals that limit broader use. SpyRL is presented as an advance that converts an asymmetric spy-identification game into simple rule-based rewards suitable for training on open-ended tasks.
A. Related Work
Multi-Agent Debate and Interaction.
The idea of using debate as a mechanism for AI alignment was proposed by [17]. Since then, multi-agent debate has been shown to improve factuality and reasoning ([18]), evaluation quality ([19]), divergent thinking ([20]), and truthfulness ([21]) in LLMs. Social deduction games have also been used to train LLMs with multi-agent RL ([22]). However, these approaches primarily operate at inference time or target specific game performance; few convert multi-agent outcomes into training rewards for general open-domain tasks. By contrast, SpyRL uses an information-asymmetric adversarial game with rule-based outcomes as a verifiable RL training signal for arbitrary open-domain tasks.
Self-Play for LLMs.
Self-play has driven breakthroughs from AlphaGo ([23, 24]) to AlphaZero ([25]) and OpenAI Five ([26]), and asymmetric self-play creates automatic curricula ([27]). In LLMs, self-play has been adapted for self-improvement ([28, 6]) and reasoning. Proposer-solver frameworks such as Absolute Zero ([12]), R-Zero ([13]), and Tool-R0 ([29]) jointly evolve task generators and solvers, often achieving strong performance without external data ([30]). Multi-agent self-play methods, including SPIRAL ([15]), SPICE ([31]), SPAG ([32]), and SPELL ([33]), target reasoning through competitive games. Concurrently, Vision-Zero ([34]) extends the same "Who Is the spy" game structure to vision-language models, enabling label-free self-improvement from arbitrary image inputs. These methods primarily focus on verifiable domains (math/code) or use game-specific outcomes as indirect signals for general capabilities, risking degeneration under prolonged training ([16, 35]). SpyRL extends self-play to non-verifiable domains via an information-asymmetric game explicitly designed so that game success requires target-task proficiency, yielding verifiable training signals without answer-level verification.
RL Beyond Verifiable Domains.
RLHF ([3, 36]) and its variants ([37, 38, 4]) replace deterministic verification with preference signals, while LLM-as-a-Judge ([5]), rubric-based rewards ([39]), and self-rewarding approaches ([6]) approximate verifiable feedback for open-ended tasks. Process reward models ([40, 41]) provide step-level feedback but require problems with deterministic answers. Concurrently, Writing-Zero ([42]) bridges non-verifiable tasks via generative reward models. Extending RL beyond verifiable domains thus requires either expensive human preferences, model-generated judgments that introduce bias, or ground-truth answers that limit domain coverage. SpyRL takes a different approach: it transforms the quality objective into a verifiable identity-recognition problem where spy identity provides deterministic ground truth, yielding rule-based rewards without learned verifiers.
B. Extended Related Work
Multi-Agent Debate and Interaction for LLMs.
Multi-agent systems have a rich history in game AI, from TD-Gammon ([43]) and AlphaGo ([23, 24]) to AlphaStar ([44]) and emergent tool use from multi-agent autocurricula ([45]). In language, multi-agent interaction has been explored through emergent communication ([46]) and strategic reasoning in Diplomacy ([47]). The idea of using debate as a mechanism for AI alignment was proposed by [17]. Since then, multi-agent debate has been shown to improve factuality and reasoning in LLMs ([18]), enhance evaluation quality via deliberation ([19]), encourage divergent thinking ([20]), and produce more truthful answers through debate ([21]). Social deduction games such as Among Us have also been used to train LLMs with multi-agent RL ([22]), and multi-agent meta-reasoning has been explored in REMA ([48]). However, these approaches primarily operate at inference time to improve output quality, or target performance within specific game environments; few convert multi-agent interaction outcomes into training rewards for general open-domain tasks. By contrast, SpyRL formulates multi-agent interaction as an information-asymmetric adversarial game with rule-based outcomes, converting the interaction signal into a verifiable RL training reward applicable to arbitrary open-domain tasks.
Self-Play Training for LLMs.
Self-play has driven sustained capability scaling from AlphaZero ([25]) to OpenAI Five ([26]), and asymmetric self-play has been shown to create powerful automatic curricula ([27]). In the LLM era, RL with verifiable rewards (RLVR) at scale, exemplified by DeepSeek-R1 ([2]), OpenAI o1 ([1]), and Kimi k1.5 ([49]), has demonstrated that rule-based rewards can unlock chain-of-thought reasoning ([50]). Self-play has been adapted for LLM self-improvement through SPIN ([28]) and Self-Rewarding Language Models ([6]), while SeRL ([51]) combines self-instruction with self-rewarding under limited data. A series of recent proposer-solver frameworks, including Absolute Zero ([12]), R-Zero ([13]), Self-Questioning LMs ([52]), and Tool-R0 ([29]), jointly evolve task generators and solvers via self-play, often achieving strong performance without external data ([53, 30]). Multi-agent self-play has also been explored for reasoning: SPIRAL ([15]) and SPICE ([31]) leverage competitive games and corpus environments respectively, SPAG ([32]) uses adversarial taboo, SPELL ([33]) targets long-context evolution, SPC ([54]) evolves critics via adversarial games, and Prover-Verifier Games ([55]) improve output legibility through adversarial training. Concurrently, Vision-Zero ([34]) extends the "Who Is the spy" game structure to vision-language models, demonstrating that the same self-play paradigm enables label-free VLM self-improvement from arbitrary image inputs. However, these methods primarily target verifiable domains (math/code) or use game-specific outcomes as indirect signals for general capabilities, which limits applicability to open-ended tasks where objectives resist simple rule-based evaluation. Recent analyses further show that prolonged self-play risks degeneration and performance collapse ([16, 35]). SpyRL extends self-play to non-verifiable domains by designing an information-asymmetric adversarial game in which game success directly requires target-task proficiency, yielding verifiable training signals without answer-level verification.
Reinforcement Learning Beyond Verifiable Domains.
RLHF ([3, 36]) and its variants, including Constitutional AI ([37]), RLAIF ([38]), and DPO ([4]), replace deterministic verification with learned or model-generated preference signals, but these approaches require expensive preference data or an external reward model whose quality bounds the learning. LLM-as-a-Judge ([5]) and rubric-based rewards ([39]) attempt to approximate verifiable feedback for open-ended tasks, while self-rewarding approaches ([6]) let the model serve as its own judge, coupling actor and evaluator capabilities. Process reward models ([40, 41]) provide step-level feedback but require problems with deterministic ground-truth answers. Curriculum design ([56]), unsupervised environment generation ([57]), and scalable RL training systems ([58]) improve training efficiency but do not address the fundamental challenge of reward construction for unverifiable tasks. Concurrently, Writing-Zero ([42]) bridges non-verifiable creative writing tasks and verifiable rewards via principle-based generative reward models, and AlphaProof ([59]) demonstrates that grounded RL with formal verification can produce complex mathematical reasoning strategies. A common thread across these approaches is that extending RL beyond verifiable domains requires either expensive human preference data, model-generated quality judgments that introduce systematic bias, or ground-truth answers that limit domain coverage. SpyRL takes a fundamentally different approach: rather than approximating task quality via human or model judgments, it transforms the quality objective into a verifiable identity-recognition problem through multi-agent competition. Because the spy identity is assigned by the environment, the correctness of detection votes is deterministically verifiable, yielding rule-based rewards that require neither external annotations nor learned verifiers.
C. Implementation Details
C.1 Role-Advantage Estimation
In the performing stage, the spy player and the civilian players operate under structurally asymmetric information: the spy observes the degraded input $g(x)$ while civilians observe the full input $x$. This asymmetry induces systematically different expected reward distributions across the two roles, even when the underlying policy is identical. Directly using the raw performing rewards $r_u^P$ and $r_{c_j}^P$ for policy optimization would therefore conflate role-induced reward differences with genuine performance differences, leading to biased gradient estimates.
Following [15], we adopt Role-Advantage Estimation (RAE) to decouple role-specific reward baselines from the optimization signal. We maintain two exponential moving average (EMA) baselines, one for each role:
$ b_u \leftarrow \alpha , b_u + (1-\alpha) , r_u^P, \qquad b_c \leftarrow \alpha , b_c + (1-\alpha) , \frac{1}{n_c}\sum_{j=1}^{n_c} r_{c_j}^P,\tag{12} $
where $\alpha \in [0, 1)$ is the EMA decay rate and both baselines are initialized to zero. The baseline $b_u$ tracks the expected reward for the spy role, while $b_c$ tracks the expected reward for the civilian role. The role-calibrated advantages for the performing stage are then computed as:
$ A_u^P = r_u^P - b_u, \qquad A_{c_j}^P = r_{c_j}^P - b_c, \quad j = 1, \dots, n_c.\tag{13} $
These advantages $A_k^P$ (for $k \in {u} \cup \mathcal{C}$) replace the raw rewards in the performing-stage policy gradient Equation (10 in the main text). By subtracting role-specific baselines, RAE ensures that the gradient signal reflects how well a player performed relative to the typical outcome for its assigned role, rather than being confounded by the inherent difficulty difference between playing as the spy versus a civilian. This prevents the optimization from systematically favoring one role over the other and stabilizes training throughout the alternating optimization process.
::: {caption="Table 11: Parameter settings for state transitions. We report the thresholds used for transitions between the Detection and Performing states, together with the minimum dwell time."}
:::
C.2 Alternating Optimization Strategy
Pure self-play frameworks often suffer from local equilibria or knowledge saturation, where models merely exploit current game distributions rather than exploring novel reasoning paths. To sustain a challenging learning environment and ensure continuous co-evolution, SpyRL employs a dynamic, two-stage alternating training scheme between the Detection and Performing stages.
Intuitively, if the detection stage easily identifies the spy player, the performing policy (specifically the spy's ability to blend in) is under-optimized and needs improvement. Conversely, if detection frequently fails or the model abstains, the detection policy is saturated and requires training. To formalize this, we monitor the detection performance over a mini-batch $\mathcal{B}_t$ at iteration $t$. We calculate the average identification accuracy $\mathrm{acc}_t$ and the uncertainty ("N/A") rate $\mathrm{na}_t$. To prevent noisy gradient updates from triggering premature stage switches, we apply an exponential moving average (EMA) with a smoothing factor $\rho \in [0, 1)$ to obtain stable estimates $\bar{\mathrm{acc}}_t$ and $\bar{\mathrm{na}}_t$.
Let $m_t \in {0, 1}$ denote the active training phase, where $m_t=1$ activates the Performing stage and $m_t=0$ activates the Detection stage. We govern the phase transitions using a set of hysteresis thresholds ($\tau^{\uparrow}{\mathrm{acc}}, \tau^{\uparrow}{\mathrm{err}}, \tau^{\uparrow}{\mathrm{na}}, \tau^{\downarrow}{\mathrm{na}}$):
$ \begin{aligned} \textbf{Detection} \rightarrow \textbf{Performing} \ (m_{t+1}=1):&\quad \text{if } m_t=0 \land \bar{\mathrm{acc}}t \ge \tau^{\uparrow}{\mathrm{acc}} \land \bar{\mathrm{na}}t \le \tau^{\downarrow}{\mathrm{na}}, \quad\text{(a)} \ \textbf{Performing} \rightarrow \textbf{Detection} \ (m_{t+1}=0):&\quad \text{if } m_t=1 \land \Big(1-\bar{\mathrm{acc}}t \ge \tau^{\uparrow}{\mathrm{err}} \lor \bar{\mathrm{na}}t \ge \tau^{\uparrow}{\mathrm{na}}\Big). \quad\text{(b)} \end{aligned}\tag{14} $
If neither condition is met, the phase remains unchanged ($m_{t+1} = m_t$). To further avoid training chattering, we enforce a minimum dwell time of $K_{\min}$ updates per phase. The threshold values used in all experiments are listed in Table 11. Under this gating mechanism, gradients are exclusively routed to the active module.
This alternating paradigm provides two critical benefits: (1) It prevents the model from stagnating in a strategic equilibrium by dynamically switching stages based on real-time saturation signals, ensuring continuous adversarial improvement. (2) It introduces a stable supervision signal derived from verifiable game mechanics, preventing common multi-agent pitfalls such as role collapse or divergence.
C.3 Training Hyperparameters
We optimize our model using the Group Relative Policy Optimization (GRPO) algorithm, implemented via the verl training framework. The base model for our actor and reference policies is Qwen/Qwen3-4B-Instruct-2507. Training was conducted on a single node equipped with 8 GPUs.
\begin{tabular}{lc}
\toprule
\textbf{Hyperparameter} & \textbf{Value} \\
\midrule
\multicolumn{2}{l}{\textit{Algorithm \& Optimization}} \\
RL Algorithm & GRPO \\
Learning Rate & $1 \times 10^{-6}$ \\
Prompts per Batch & 128 \\
GRPO Rollouts per Prompt & 8 \\
Effective Batch Size & 1024 \\
PPO Mini-batch Size & 128 \\
Micro-batch Size (per GPU) & 2 \\
KL Penalty Coefficient ($\beta$) & 0.001 \\
KL Loss Type & Low-variance KL \\
Training Iterations & 100 \\
\midrule
\multicolumn{2}{l}{\textit{Length Constraints \& Environment}} \\
Max Prompt Length & 12, 288 \\
Max Response Length & 4, 096 \\
Max Model Length & 16, 384 \\
Number of Players & 5 \\
Number of Rounds & 1 \\
\midrule
\multicolumn{2}{l}{\textit{System \& Memory}} \\
Hardware & 1 Node $\times$ 8 GPUs \\
Rollout Engine & vLLM (TP=8) \\
Gradient Checkpointing & True \\
Reference Model Offload & True \\
Actor Model Offload & False \\
\bottomrule
\end{tabular}
Optimization and Algorithm Settings.
For the GRPO algorithm, we sample a group of $n=8$ responses per prompt during the rollout phase. With 128 prompts per batch and 8 rollouts each, this yields an effective batch size of 1024 samples. The actor model is trained with a learning rate of 1 x 10^-6 for 100 training iterations. The PPO mini-batch size is set to 128, with a micro-batch size of 2 per GPU. To prevent the policy from deviating excessively from the reference model, we apply a low-variance KL divergence penalty with a coefficient of $0.001$. The entropy coefficient is set to 0.
Generation and Rollout.
During the interactive rollout phase, we utilize the vLLM engine to accelerate generation, setting tensor model parallelism (TP) to 8 and restricting GPU memory utilization to 0.45 to leave sufficient memory for the training weights. We allow a maximum prompt length of 12, 288 tokens and generate responses up to 4, 096 tokens, bounded by a total maximum model length of 16, 384 tokens. The game environment is configured for 5 players interacting over 1 round.
Memory Management and System Configurations.
To manage GPU memory efficiently during the reinforcement learning process, we enable gradient checkpointing for the actor model. Furthermore, we employ Fully Sharded Data Parallel (FSDP). Specifically, the reference model's parameters are offloaded to the CPU (param_offload=True) to save VRAM, while the actor model's parameters and optimizer states remain on the GPU to maximize training throughput.
Table 12 summarizes the key hyperparameter configurations used in our experiments.
The complete launch script and detailed configuration flags are provided below for reproducibility:
python3 -m verl.trainer.main_ppo \
algorithm.adv_estimator=grpo \
data.train_batch_size=128 \
data.train_max_samples=100000000 \
data.max_prompt_length=12288 \
data.max_response_length=3762 \
data.filter_overlong_prompts=True \
+data.num_players=5 \
+data.num_rounds=1 \
+data.prompt_max_tokens=128 \
actor_rollout_ref.rollout.agent.default_agent_loop=writingprompts_two_player \
custom_reward_function.name=compute_score \
+custom_reward_function.reward_kwargs.max_debug_prints=4 \
actor_rollout_ref.model.path=Qwen/Qwen3-4B-Instruct-2507 \
actor_rollout_ref.actor.optim.lr=1e-6 \
actor_rollout_ref.actor.use_torch_compile=False \
actor_rollout_ref.model.use_remove_padding=True \
actor_rollout_ref.actor.ppo_mini_batch_size=128 \
actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=2 \
actor_rollout_ref.actor.use_kl_loss=True \
actor_rollout_ref.actor.kl_loss_coef=0.001 \
actor_rollout_ref.actor.kl_loss_type=low_var_kl \
actor_rollout_ref.actor.entropy_coeff=0 \
actor_rollout_ref.model.enable_gradient_checkpointing=True \
actor_rollout_ref.actor.fsdp_config.param_offload=False \
actor_rollout_ref.actor.fsdp_config.use_torch_compile=False \
actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \
actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=2 \
actor_rollout_ref.rollout.tensor_model_parallel_size=8 \
actor_rollout_ref.rollout.name=vllm \
actor_rollout_ref.rollout.gpu_memory_utilization=0.45 \
actor_rollout_ref.rollout.enforce_eager=True \
+actor_rollout_ref.rollout.engine_kwargs.vllm.compilation_config.cudagraph_mode=NONE \
+actor_rollout_ref.rollout.engine_kwargs.vllm.compilation_config.use_inductor=False \
actor_rollout_ref.rollout.agent.num_workers=1 \
actor_rollout_ref.rollout.max_num_seqs=128 \
actor_rollout_ref.rollout.max_num_batched_tokens=8192 \
actor_rollout_ref.rollout.max_model_len=16384 \
actor_rollout_ref.rollout.response_length=4096 \
actor_rollout_ref.rollout.n=8 \
actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=2 \
actor_rollout_ref.ref.use_torch_compile=False \
actor_rollout_ref.ref.fsdp_config.use_torch_compile=False \
actor_rollout_ref.ref.fsdp_config.param_offload=True \
algorithm.use_kl_in_reward=False \
trainer.critic_warmup=0 \
trainer.val_before_train=False \
+trainer.training_phase=interactive \
+trainer.interactive_cycle_length=1 \
trainer.logger='["console"]' \
trainer.n_gpus_per_node=8 \
trainer.nnodes=1 \
trainer.save_freq=5 \
trainer.test_freq=-1 \
trainer.total_epochs=1 $@
C.4 Baselines & Metrics
To rigorously evaluate the effectiveness of our approach, we benchmark against state-of-the-art self-play methodologies and employ a comprehensive evaluation protocol combining automated metrics with robust LLM-as-a-judge A/B testing. Prompt templates for the two game stages are provided in Appendix C.5.
Baselines.
We compare our model against two leading proposer-solver self-play frameworks designed for large language models:
- R-Zero ([13]): A state-of-the-art framework that leverages a self-contained reinforcement learning loop. In this paradigm, a proposer model generates candidate solutions or trajectories, while a solver (or reward model) evaluates them based on internal logic or rule-based constraints. This method eliminates the need for external human-annotated preference data by iteratively optimizing the proposer against the solver's feedback.
- Absolute Zero ([12]): An advanced pure self-play framework inspired by zero-shot bootstrapping principles. It relies on iterative self-refinement where the model acts as both the generator and the verifier. Absolute Zero focuses on discovering optimal strategies entirely from the base model's inherent capabilities, providing a strong baseline for self-taught reasoning and strategic generation.
By comparing against these frameworks, we aim to demonstrate whether our specific multi-agent interactive training offers superior strategic adaptation and generation quality compared to standard single-agent or dual-agent proposer-solver loops.
Automatic Metrics.
Quantitative evaluation is first conducted using task-specific automated metrics. Depending on the exact nature of the game rounds, these metrics measure structural compliance, such as format adherence (e.g., successful extraction of the required \boxed formatting), exact match accuracy for the detection stage (whether the spy is correctly identified), and basic linguistic metrics (e.g., word count constraints and repetition penalties). These rule-based metrics provide an objective baseline for the model's fundamental instruction-following capabilities.
GPT-4o A/B Testing and Position Bias Mitigation.
Because automated metrics fall short in assessing open-ended creativity, strategic depth, and narrative coherence (especially in the story-writing phase), we utilize GPT-4o as an impartial judge to conduct pairwise A/B testing. We compare the responses generated by our trained model directly against those from the base model and the baselines.
A well-known challenge in LLM-as-a-judge evaluation is position bias (also known as order bias), where the evaluator disproportionately favors either the first or the second option presented in the prompt, regardless of actual quality. To rigorously mitigate this bias and ensure statistical significance, we implement a swapped-order evaluation protocol:
- For every comparison between Model A and Model B on a given prompt, we query GPT-4o twice.
- In the first query, Model A's output is presented as "Candidate 1" and Model B's as "Candidate 2".
- In the second query, the order is strictly reversed: Model B is presented as "Candidate 1" and Model A as "Candidate 2".
We aggregate the pairwise results conservatively: Model A is only awarded a Win if it is preferred in both permutations, or if it wins in one permutation and ties in the other. If Model A wins in the first query but loses in the swapped query (or vice versa), the result is recorded as a Tie. This strict aggregation heavily penalizes position bias and ensures that any reported win rate reflects a genuine, robust preference for the generated content's quality.
C.5 Prompt Design and Configurations
In our framework, the interactions are driven by two carefully engineered prompts corresponding to the two main phases of the game: the Performing Stage and the Detection Stage. These prompts are designed not merely to instruct the models, but to induce strategic reasoning, enforce high-quality text generation, and ensure robust automated parsing.
C.5.1 Performing Stage Prompt Design
The prompt for the Performing Stage acts as the system instruction for the agents generating the stories. Its key design advantages include:
- Asymmetric Role Awareness: The prompt dynamically injects variables such as
{role_info}androle_instruction, enabling the same base template to function for both civilian players and the spy. It explicitly emphasizes the information gap (the spy sees a blank prompt), establishing the core tension of the game. - Strategic Chain-of-Thought (CoT): We require the agent to "conduct your own thinking process" before outputting the final answer. For the spy, this encourages reasoning about what the hidden prompt might be based on previous context; for civilian players, it encourages planning the narrative structure.
- High-Quality Constraints: To prevent standard, cliche LLM responses, the prompt provides explicit writing criteria (e.g., "include subtext, a turning point, " "fresh imagery"). This forces the model to generate diverse and literary outputs rather than generic summaries.
- Strict Formatting: By mandating the exact prefix
"Answer:"and restricting bullet points, we ensure the output is easily parsed by our automated evaluation pipeline.
game_context = f"""You are PLAYER player_id of the game "Who's the Odd One Out?". This is ROUND round_num. Your task is to write ONE short creative story based on the writing prompt.
role_info
Game Instructions:
- There are num_players players in total. Only one player (the spy) received NO writing prompt, while all other players received the SAME writing prompt.
- The player who is voted as the odd player will be severely punished, so you should convince others that you are not the odd player.
- The spy sees nothing (blank), while all other players see the same writing prompt.
role_instruction
What you should do:
1. Compare your writing prompt (or lack thereof) with all previous stories to understand what others are writing about.
2. If you are NOT the spy:
1) Identify the core theme, mood, and implied constraints in the prompt.
2) Write a short story that clearly matches the theme and is NOT generic.
3) Make it deep: include subtext, a turning point, or a meaningful implication.
4) Keep it coherent: clear setup -> development -> payoff, with smooth transitions.
5) Be creative: fresh imagery, non-obvious choices, clever structure or constraints.
3. If you are the spy:
1) Infer the likely theme/prompt by reading previous players' stories.
2) Write a plausible story that fits what others are writing about.
3) Your story can be slightly more general, but it should still feel thematic, coherent, and creative -- not obviously off-topic.
Fixed template you MUST follow:
- Story template: Write ONE continuous short story in plain paragraphs (no bullet points), with 300-500 words (strict). Include a clear turning point and a payoff.
Required output format:
First, conduct your own thinking process.
Then, start a new line with exactly "Answer:" and write exactly ONE short story following the Story template after it (may span multiple paragraphs).
prompt_display
"""
C.5.2 Detection Stage Prompt Design
The Detection Stage prompt transforms the LLM into a critical evaluator. To minimize hallucination and subjective judging, we incorporate the following design features:
- Multi-dimensional Evaluation Rubric: Instead of simply asking "who is the spy, " the prompt provides 5 specific dimensions of anomaly detection (e.g., Off-theme, Shallow, Consistency with others). This grounds the model's reasoning in concrete evidence.
- Uncertainty Calibration: By explicitly permitting the output
\boxed{N/A}, we allow the model to express uncertainty. This prevents forced guesses when evidence is insufficient and mimics human hesitation, allowing the model to continue reasoning in subsequent iterations. - Efficiency and Length Control: The prompt imposes a strict token limit (
<= 2000 tokens) and forbids excessive brainstorming, preventing the model from entering infinite reasoning loops and saving computational overhead. - Robust Extraction Standard: Adopting a common standard from mathematical reasoning tasks in RL, the final decision must be wrapped in
\boxed. This guarantees deterministic extraction of the predicted player ID during training or evaluation.
decision_context = f"""You are analyzing a game of "Who's the Odd One Out?" where num_players players have just written short stories based on a writing prompt. Your task is to identify which player is the spy (the one who received NO writing prompt) based on their story.
Game Setup:
- There are num_players players in total. Only one player (the spy) received NO writing prompt, while all other players received the SAME writing prompt.
- The spy sees nothing (blank prompt).
- Each player wrote a short story trying to match the prompt while avoiding suspicion.
- The spy tries to blend in by inferring the theme from others and writing something that seems consistent.
- Non-spy players try to write stories that are thematically aligned, meaningful, coherent, and creative.
Your task:
Analyze the writing prompt and all the stories and determine which player is most likely the spy. Your primary strategy is to judge who likely had access to the prompt.
Look for:
1. **Off-theme / mismatch**: The story does not match the prompt's theme, constraints, or implied setting.
2. **Shallow / not meaningful**: Lacks subtext, turning point, or deeper implication.
3. **Weak narrative craft**: Straight-line narration, poor coherence, no payoff, abrupt transitions.
4. **Low creativity**: Generic style, cliche patterns, unimaginative choices.
5. **Consistency with others**: Non-spy players tend to converge on the same prompt-driven theme; a spy may drift or imitate superficially.
Efficiency constraints (IMPORTANT):
- Do a fast check for each story (on-theme? meaningful? coherent? creative?) and pick the single most suspicious player.
- If you cannot immediately determine who the mole is, answer \\boxed{{N/A}} first and then continue thinking.
- Do NOT brainstorm. The entire output must be <= 2000 tokens.
The writing prompt below is what non-spy players see. Compare each story against this reference:
[Reference Writing Prompt]
writing_prompt
Required output format:
First, conduct your private reasoning -- may include suspicions, probabilities, evidence analysis, etc.
Then, put your final answer (PLAYER_NUMBER or N/A) inside \\boxed{}. If you are uncertain, you can answer N/A.
Example answer: \\boxed{{1}}; \\boxed{{2}}; \\boxed{{3}}; \\boxed{{N/A}}.
Hard limit: The entire output must be <= 2000 tokens.
All Stories from the Story-writing Stage:
all_stories"""
::: {caption="Table 13: A/B test evaluation against the untrained base model on summarization benchmarks. Each row reports the GPT-4o win rate (%) of a trained model against its own untrained backbone, so 50% denotes parity and the untrained-model rows act as a calibration reference. Avg. is the mean over the five benchmarks. Shaded rows are SpyRL."}
:::
::: {caption="Table 14: A/B test evaluation against the untrained base model on writing benchmarks. Each row reports the GPT-4o win rate of a trained model against its own untrained backbone, so 50% denotes parity. Higher is better."}
:::
D. Additional Experiments
This section presents additional experiments that complement the main results from different perspectives. These experiments further examine the effectiveness, training stability, cross-task transferability, evaluation robustness, and human alignment of SpyRL, providing a more comprehensive validation of the proposed framework.
D.1 A/B Evaluation against the Untrained Base Model
The A/B evaluations in the main text are anchored on the competing method: each row of Table 1 and Table 2 compares SpyRL directly against one opponent. That presentation answers which method a reader should prefer, but it does not show how much each baseline gains from its own self-evolution procedure. For completeness, Table 13 and Table 14 therefore report the complementary view for summarization and creative writing, in which every method is compared against its own untrained backbone under the same swapped-order protocol. A win rate of $50%$ denotes parity, and the untrained-model rows act as a calibration reference.
Three observations follow. First, the untrained backbones score close to chance against themselves, with average win rates of $51.7%$ and $51.8%$ on summarization and overall win rates between $51.0%$ and $51.8%$ on creative writing. This confirms that the swapped-order aggregation described in Appendix C.4 removes most of the position bias, so deviations from $50%$ elsewhere in the table reflect genuine quality differences.
Second, the two self-evolution baselines move the needle very little on either task. On summarization, R-Zero averages $51.9%$ and $51.5%$ for Qwen3-4B and Qwen3-8B, i.e. statistically indistinguishable from no training at all, while Absolute Zero reaches $61.5%$ and $59.0%$. On creative writing the gap is starker: Absolute Zero attains overall win rates of only $54.0%$ and $55.2%$ with Qwen3-4B and $56.4%$ and $58.1%$ with Qwen3-8B, and R-Zero is preferred less often than the untrained Qwen3-4B ($48.8%$ and $46.5%$), meaning that its training actively degrades creative writing. SpyRL, by contrast, averages $73.9%$ and $75.4%$ on summarization and attains overall win rates of $81.3%$ and $75.1%$ with Qwen3-4B and $76.5%$ and $78.1%$ with Qwen3-8B on creative writing.
Third, this contrast follows directly from how the three methods obtain their reward. R-Zero and Absolute Zero require verifiable solver feedback to calibrate task difficulty, which is unavailable for open-ended generation, so their proposer–solver loop has nothing reliable to optimize against. SpyRL instead derives its reward from the environment-assigned spy identity, which remains well defined regardless of whether the target task admits a verifier. Together with the main-text results, this confirms that the gains of SpyRL are not an artifact of the choice of evaluation anchor.
D.2 Evaluation on More Challenging Reasoning Benchmarks
The reasoning benchmarks in the main experiments span a wide difficulty range, but the hardest of them (AIME 2024 and AIME 2025) contain only 30 problems each, so we additionally evaluate on three larger and more demanding benchmarks: AMC, Olympiad-Bench, and SuperGPQA. We reuse the Qwen3-4B checkpoints trained under our mathematical reasoning setting without any further fine-tuning, and compare Qwen3-4B, R-Zero, Absolute Zero, and SpyRL. We clarify that the Absolute Zero baseline in this work is not the code-oriented checkpoint released with the original paper. Instead, we reimplement and train Absolute Zero with the same Qwen3-4B backbone under our math-domain setting, ensuring a controlled and fair comparison across methods.
As shown in Table 15, SpyRL achieves the best performance on all three benchmarks, reaching 55.0 on AMC, 42.7 on Olympiad-Bench, and 30.1 on SuperGPQA, compared with 47.5, 34.8, and 25.4 for the untrained backbone. Averaged over the three benchmarks this is a gain of 6.7 points, against 1.6 for R-Zero and 3.6 for Absolute Zero. SpyRL is also the only method that improves on every benchmark: R-Zero falls below the untrained backbone on AMC, indicating that its proposer–solver curriculum does not transfer to competition-style problems.
These results demonstrate that the improvements of SpyRL extend beyond the benchmarks reported in the main text and remain effective on more difficult reasoning problems. We attribute this advantage to the information-asymmetric self-play mechanism: incomplete derivations, overlooked conditions, and unsupported assumptions are more likely to expose the spy player during detection, allowing these reasoning deficiencies to be converted into verifiable training signals. Consequently, SpyRL encourages the model to produce more complete, rigorous, and reliable reasoning processes.
::: {caption="Table 15: Results on harder reasoning benchmarks. We report accuracy (%) of Qwen3-4B and its self-evolution variants on AMC, Olympiad-Bench, and SuperGPQA, using the same math-domain checkpoints as [@tab:main_results]. Avg. is the mean over the three benchmarks and Delta the gain over the untrained backbone. Absolute Zero is reimplemented on the same backbone under our math-domain setting rather than taken from the released code-oriented checkpoint, so that all methods are trained comparably. The shaded row is SpyRL."}
:::
D.3 Joint versus Alternating Two-Stage Optimization
The performing and detection stages of SpyRL are mutually dependent: the detector's votes determine the rewards used to optimize the performer, while the outputs generated by the performer constitute the training inputs for the detector. A straightforward implementation is to jointly update both stages within every training epoch. However, this strategy introduces a tightly coupled and highly non-stationary optimization process, particularly during the early stages of training. At initialization, the detector has limited ability to identify the spy player, while both civilian and spy players produce relatively weak task outputs. Consequently, the detector's votes may not reliably reflect differences in performing quality, resulting in noisy or misleading rewards for the performing stage. Updating the performer with such inaccurate rewards can further degrade its outputs, which in turn provides the detector with less informative training examples.
To alleviate this issue, SpyRL adopts an alternating optimization strategy in which only one stage is updated during each training epoch while the other stage remains fixed. In particular, strengthening the detector before using its voting outcomes to optimize the performer provides a more reliable reward signal for the performing stage. Once the detector can meaningfully distinguish outputs generated under complete and degraded information, the performer is encouraged to produce more rigorous and strategically convincing responses. These improved outputs subsequently increase the difficulty of identity detection and provide more informative examples for further detector training. Alternating optimization therefore decomposes the coupled learning problem into more stable stage-wise updates and allows the performing and detection capabilities to progressively shape each other.
The resulting performance on five mathematical reasoning benchmarks is reported in Table 16. Joint training fails to improve the base model and substantially degrades performance on several benchmarks. For example, accuracy decreases from 84.5 to 76.8 on GSM8K, from 68.2 to 53.1 on Math500, and from 42.3 to 33.1 on Minerva. Its average score across the five benchmarks drops from 42.4 to 35.3.
In contrast, alternating optimization consistently improves the model across all evaluated benchmarks. It increases GSM8K accuracy from 84.5 to 93.4 and Math500 accuracy from 68.2 to 79.5. The improvement is particularly pronounced on AIME 2025, where accuracy rises from 6.7 to 20.0. Averaged across the five benchmarks, alternating training achieves a score of 50.8, corresponding to an improvement of 8.4 points over the base model and 15.5 points over joint training. These results support our analysis that simultaneously updating the two mutually dependent stages leads to inaccurate credit assignment and unstable learning, whereas alternating optimization provides sufficiently reliable intermediate signals for sustained co-evolution between the performer and detector.
::: {caption="Table 16: Comparison of joint and alternating two-stage optimization. We report the accuracy of Qwen3-4B. Joint training updates the performer and detector simultaneously, whereas alternating training updates only one stage in each epoch."}
:::
D.4 Evaluation with an Alternative LLM Judge
To examine whether the improvements of SpyRL are robust to the choice of automatic evaluator, we repeat the A/B evaluations on summarization and creative writing using Gemini-3.5-Flash as an alternative judge. We follow the same evaluation protocol as in the main experiments: for each test instance, the evaluator compares the anonymized outputs of the SpyRL-trained model and its corresponding original Qwen3 model. We also aggregate judgments obtained under swapped response orders to mitigate position bias. The reported value is the proportion of comparisons in which the evaluated model is preferred over the original model.
As shown in Table 17, SpyRL consistently improves summarization performance across all five benchmarks. The Qwen3-4B model trained with SpyRL obtains an average win rate of $79.8%$, compared with $52.3%$ for the original model, while the corresponding Qwen3-8B model achieves an average win rate of $76.1%$, compared with $52.0%$ for its base model. The improvements are consistent across GovReport, Multi-News, QMSum, VCSum, and SAMSum, indicating that the gains are not limited to a particular summarization domain.
A similar trend is observed for creative writing in Table 18. SpyRL substantially outperforms the original models across novelty, emotion, coherence, consistency, and overall quality on both WritingPrompts and WritingBench. In particular, Qwen3-4B achieves overall win rates of $81.8%$ and $81.4%$ on the two benchmarks, while Qwen3-8B achieves $77.8%$ and $77.0%$, respectively. These results are consistent with the GPT-4o-based evaluation in the main experiments, demonstrating that the observed improvements remain stable under a different LLM judge and are unlikely to arise from evaluator-specific preferences.
::: {caption="Table 17: Gemini-3.5-Flash A/B evaluation on summarization benchmarks. We report the win rates of the original Qwen3 models and their SpyRL-trained counterparts against the corresponding base model across five summarization benchmarks. Higher is better."}
:::
::: {caption="Table 18: Gemini-3.5-Flash A/B evaluation on creative writing benchmarks. We report win rates across five level quality on WritingPrompts and WritingBench."}
:::
D.5 Agreement between GPT-4o and Human Evaluation
To examine the reliability of GPT-4o as an automatic evaluator, we measure its agreement with the human annotations collected in our creative-writing evaluation. Treating the human judgments as ground-truth labels, we formulate the pairwise preference for SpyRL as a binary classification problem and compute the precision and recall of GPT-4o for each evaluation dimension. Here, precision measures the proportion of samples judged as wins for SpyRL by GPT-4o that are also preferred by human evaluators, while recall measures the proportion of human-preferred SpyRL samples that are correctly identified by GPT-4o.
As shown in Table 19, GPT-4o exhibits strong agreement with human judgments across all five dimensions. Precision ranges from $85.7%$ to $91.0%$, while recall ranges from $79.4%$ to $93.8%$. In particular, GPT-4o achieves strong performance on novelty, emotion, and overall quality, with an overall precision of $91.0%$ and recall of $93.8%$. Although the recall for consistency is relatively lower at $79.4%$, the results remain consistently high overall. These findings indicate that GPT-4o provides a reliable approximation of human preferences and support its use as an automatic evaluator for comparing the quality of open-ended creative-writing outputs.
::: {caption="Table 19: Agreement between GPT-4o and human evaluation. Treating human judgments as ground-truth labels, we report the precision and recall of GPT-4o in identifying pairwise wins for SpyRL across five creative-writing evaluation dimensions. Higher is better."}
:::
References
Section Summary: This section compiles dozens of academic citations, mostly recent arXiv preprints and conference papers, that explore how large language models can be trained or improved through reinforcement learning, self-play, and multi-agent interactions. The references cover foundational work on unsupervised learning and game mastery alongside newer studies on reasoning enhancement, reward modeling, and AI safety without relying on human data. Together they trace a research thread from early representation learning to current efforts at building self-improving AI systems.
[1] OpenAI (2024). Openai o1 system card. arXiv preprint arXiv:2412.16720.
[2] DeepSeek-AI (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv:2501.12948.
[3] Ouyang et al. (2022). Training language models to follow instructions with human feedback. Advances in neural information processing systems. 35. pp. 27730–27744.
[4] Rafailov et al. (2023). Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems. 36. pp. 53728–53741.
[5] Zheng et al. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems. 36. pp. 46595–46623.
[6] Yuan et al. (2024). Self-rewarding language models. In Forty-first International Conference on Machine Learning.
[7] Doersch et al. (2015). Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision. pp. 1422–1430.
[8] Noroozi, Mehdi and Favaro, Paolo (2016). Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision. pp. 69–84.
[9] Devlin et al. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). pp. 4171–4186.
[10] Oord et al. (2018). Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
[11] Chen et al. (2020). A simple framework for contrastive learning of visual representations. In International conference on machine learning. pp. 1597–1607.
[12] Zhao et al. (2025). Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint arXiv:2505.03335.
[13] Huang et al. (2025). R-zero: Self-evolving reasoning llm from zero data. arXiv preprint arXiv:2508.05004.
[14] Shao et al. (2024). Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
[15] Liu et al. (2025). Spiral: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning. arXiv preprint arXiv:2506.24119.
[16] Chae et al. (2025). Towards Understanding Self-play for LLM Reasoning. arXiv preprint arXiv:2510.27072.
[17] Irving et al. (2018). AI safety via debate. arXiv preprint arXiv:1805.00899.
[18] Du et al. (2024). Improving factuality and reasoning in language models through multiagent debate. In Forty-first international conference on machine learning.
[19] Chan et al. (2023). Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201.
[20] Liang et al. (2024). Encouraging divergent thinking in large language models through multi-agent debate. In Proceedings of the 2024 conference on empirical methods in natural language processing. pp. 17889–17904.
[21] Khan et al. (2024). Debating with more persuasive llms leads to more truthful answers. arXiv preprint arXiv:2402.06782.
[22] Sarkar et al. (2025). Training language models for social deduction with multi-agent reinforcement learning. arXiv preprint arXiv:2502.06060.
[23] Silver et al. (2016). Mastering the game of Go with deep neural networks and tree search. nature. 529(7587). pp. 484–489.
[24] Silver et al. (2017). Mastering the game of go without human knowledge. nature. 550(7676). pp. 354–359.
[25] Silver et al. (2018). A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science. 362(6419). pp. 1140–1144.
[26] Berner et al. (2019). Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680.
[27] Sukhbaatar et al. (2017). Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint arXiv:1703.05407.
[28] Chen et al. (2024). Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335.
[29] Acikgoz et al. (2026). Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Data. arXiv preprint arXiv:2602.21320.
[30] Kuba et al. (2025). Language self-play for data-free training. arXiv preprint arXiv:2509.07414.
[31] Liu et al. (2025). Spice: Self-play in corpus environments improves reasoning. arXiv preprint arXiv:2510.24684.
[32] Cheng et al. (2024). Self-playing adversarial language game enhances llm reasoning. Advances in Neural Information Processing Systems. 37. pp. 126515–126543.
[33] Yang et al. (2025). Spell: Self-play reinforcement learning for evolving long-context language models. arXiv preprint arXiv:2509.23863.
[34] Wang et al. (2025). Vision-zero: Scalable vlm self-improvement via strategic gamified self-play. arXiv preprint arXiv:2509.25541.
[35] Shafayat et al. (2025). Can Large Reasoning Models Self-Train?. arXiv preprint arXiv:2505.21444.
[36] Bai et al. (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862.
[37] Bai et al. (2022). Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
[38] Lee et al. (2023). Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267.
[39] Gunjal et al. (2025). Rubrics as rewards: Reinforcement learning beyond verifiable domains. arXiv preprint arXiv:2507.17746.
[40] Lightman et al. (2023). Let's verify step by step. In The twelfth international conference on learning representations.
[41] Cobbe et al. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
[42] Jia et al. (2025). Writing-zero: Bridge the gap between non-verifiable tasks and verifiable rewards. arXiv preprint arXiv:2506.00103.
[43] Tesauro, Gerald and others (1995). Temporal difference learning and TD-Gammon. Communications of the ACM. 38(3). pp. 58–68.
[44] Vinyals et al. (2019). Grandmaster level in StarCraft II using multi-agent reinforcement learning. nature. 575(7782). pp. 350–354.
[45] Baker et al. (2019). Emergent tool use from multi-agent autocurricula. In International conference on learning representations.
[46] Graesser et al. (2019). Emergent linguistic phenomena in multi-agent communication games. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). pp. 3700–3710.
[47] Meta Fundamental AI Research Diplomacy Team (FAIR)† et al. (2022). Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science. 378(6624). pp. 1067–1074.
[48] Wan et al. (2025). Rema: Learning to meta-think for llms with multi-agent reinforcement learning. arXiv preprint arXiv:2503.09501.
[49] Team, Kimi and others (2025). Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599.
[50] Wei et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems. 35. pp. 24824–24837.
[51] Fang et al. (2025). Serl: Self-play reinforcement learning for large language models with limited data. arXiv preprint arXiv:2505.20347.
[52] Chen et al. (2025). Self-questioning language models. arXiv preprint arXiv:2508.03682.
[53] Singh et al. (2023). Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585.
[54] Chen et al. (2025). Spc: Evolving self-play critic via adversarial games for llm reasoning. arXiv preprint arXiv:2504.19162.
[55] Kirchner et al. (2024). Prover-verifier games improve legibility of llm outputs. arXiv preprint arXiv:2407.13692.
[56] Bengio et al. (2009). Curriculum learning. In Proceedings of the 26th annual international conference on machine learning. pp. 41–48.
[57] Dennis et al. (2020). Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems. 33. pp. 13049–13061.
[58] Yu et al. (2025). Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476.
[59] Hubert et al. (2025). Olympiad-level formal mathematical reasoning with reinforcement learning. Nature. pp. 1–3.