Back

Decision Transformer: Reinforcement Learning via Sequence Modeling

Lili ChenKevin LuAravind RajeswaranKimin LeeAditya GroverMichael LaskinPieter AbbeelAravind SrinivasIgor Mordatch

article2021NeurIPS2,498 citations

Introduces Decision Transformer, a sequence modeling approach to offline reinforcement learning that generates actions by conditioning an autoregressive Transformer on desired returns, matching or surpassing standard baselines without using dynamic programming or policy gradients.

Listen

The article addresses challenges in offline reinforcement learning, where agents must learn effective policies from fixed datasets of past experiences without further interaction. Traditional methods relying on temporal difference learning often suffer from error propagation, value overestimation, and difficulties with sparse or delayed rewards, limiting their reliability in complex sequential decision tasks.

The article set out to evaluate whether framing reinforcement learning as a conditional sequence modeling problem, using a Transformer architecture, could produce policies that match or exceed dedicated offline RL algorithms without relying on value functions or policy gradients.

The approach involved training a causally masked GPT-style model on trajectories represented as sequences of desired returns-to-go, states, and actions drawn from offline datasets. Experiments covered discrete control in four Atari games using 1% of DQN replay data, continuous control in OpenAI Gym locomotion and reacher tasks across medium, medium-replay, and medium-expert datasets, and a Key-to-Door task requiring long-term credit assignment, with comparisons to methods such as Conservative Q-Learning, REM, and behavior cloning.

Key findings indicate that the resulting Decision Transformer model performed comparably or better than state-of-the-art baselines on most tasks, achieving gamer-normalized scores competitive with or above CQL on three of four Atari games and the highest average normalized scores across Gym settings. It excelled in sparse-reward and long-horizon credit assignment scenarios, where temporal difference methods collapsed, and demonstrated the ability to generate trajectories closely matching specified target returns, including some extrapolation beyond dataset maxima. Longer context lengths improved performance, and the model avoided the need for explicit value pessimism or behavior regularization.

These results suggest that sequence modeling can serve as a simpler, scalable alternative to conventional RL algorithms by directly leveraging advances from language modeling, potentially reducing sensitivity to reward density and improving robustness in data-limited regimes. The approach may lower barriers to applying large-scale Transformer techniques in decision-making domains while highlighting opportunities to bypass some instabilities of bootstrapped value estimation.

Actionable next steps supported by the work include integrating the model into online RL pipelines as a behavior generator alongside exploration methods, applying self-supervised pretraining on larger datasets, and testing more sophisticated embeddings for stochastic settings. Further investigation is needed to characterize failure modes in Markov decision processes and assess risks from biases in training data before broader deployment.

Limitations include evaluation on a restricted set of benchmarks and dataset sizes, with performance in very low-data regimes or real-world MDPs remaining uncertain; confidence is moderate for the reported domains given consistent multi-seed results but should be tempered for generalization beyond the tested environments.

GoogleMetaUniversity of California Berkeleyencoder-decoderlanguage modelvalue functionsDecision TransformerReinforcement LearningTransformer architecture
arXiv: 2106.01345kzl/decision-transformer
Cover for Decision Transformer: Reinforcement Learning via Sequence Modeling

Abstract

We introduce a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. This allows us to draw upon the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT. In particular, we present Decision Transformer, an architecture that casts the problem of RL as conditional sequence modeling. Unlike prior approaches to RL that fit value functions or compute policy gradients, Decision Transformer simply outputs the optimal actions by leveraging a causally masked Transformer. By conditioning an autoregressive model on the desired return (reward), past states, and actions, our Decision Transformer model can generate future actions that achieve the desired return. Despite its simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.

Topics

encoder-decoderlanguage modelvalue functionsmodel free

Keywords

Decision TransformerReinforcement LearningTransformer architectureoffline RLreturn‑to‑go conditioningcontext lengthcredit assignmentbehavior cloningsparse rewardAtari benchmark

Table of Contents

  • 1 Introduction
  • 2 Preliminaries
  • 2.1 Offline reinforcement learning
  • 2.2 Transformers
  • 3 Method
  • 4 Evaluations on Offline RL Benchmarks
  • 4.1 Atari
  • 4.2 OpenAI Gym
  • 5 Discussion
  • 5.1 Does Decision Transformer perform behavior cloning on a subset of the data?
  • 5.2 How well does Decision Transformer model the distribution of returns?
  • 5.3 What is the benefit of using a longer context length?
  • 5.4 Does Decision Transformer perform effective long-term credit assignment?
  • 5.5 Can transformers be accurate critics in sparse reward settings?
  • 5.6 Does Decision Transformer perform well in sparse reward settings?
  • 5.7 Why does Decision Transformer avoid the need for value pessimism or behavior regularization?
  • 5.8 How can Decision Transformer benefit online RL regimes?
  • 6 Related Work
  • 6.1 Offline reinforcement learning
  • 6.2 Supervised learning in reinforcement learning settings
  • 6.3 Credit assignment
  • 6.4 Conditional language generation
  • 6.5 Attention and transformer models
  • 7 Conclusion
  • References
  • A Experimental Details
  • A.1 Atari
  • A.2 OpenAI Gym
  • A.2.1 Decision Transformer
  • A.2.2 Behavior Cloning
  • A.3 Graph Shortest Path
  • B Atari Task Scores

Knowls

  1. Knowl 1 — Decision Transformer Architecture and Trajectory Representation

    model/method

    The Decision Transformer frames reinforcement learning as conditional autoregressive sequence modeling. Rather than using conventional temporal difference methods or policy gradients, the model processes trajectory rollouts containing return-to-go, state, and action tokens using a causal transformer architecture (specifically GPT).

    An environment trajectory of length TT is represented as an interleaved sequence of returns-to-go R^t\hat{R}_t, states sts_t, and actions ata_t: τ=(R^1,s1,a1,R^2,s2,a2,…,R^T,sT,aT)\tau = \left( \hat{R}_1, s_1, a_1, \hat{R}_2, s_2, a_2, \dots, \hat{R}_T, s_T, a_T \right) where the return-to-go at timestep tt represents the undiscounted sum of future rewards: R^t=∑t′=tTrt′\hat{R}_t = \sum_{t'=t}^T r_{t'}

    Each modality is projected into a shared embedding space of dimension dd using modality-specific encoders:

    • Continuous states st∈Rdss_t \in \mathbb{R}^{d_s} and continuous actions at∈Rdaa_t \in \mathbb{R}^{d_a} (or discrete action tokens) are mapped via linear layers followed by Layer Normalization. For visual inputs, states are processed using a convolutional encoder.
    • Returns-to-go R^t∈R\hat{R}_t \in \mathbb{R} are mapped via a linear layer followed by Layer Normalization.
    • A learned episodic timestep embedding et∈Rde_t \in \mathbb{R}^d is added to all three tokens corresponding to timestep tt (i.e., R^t,st,at\hat{R}_t, s_t, a_t share the same temporal position encoding ete_t).

    The embedded token sequence of the last KK timesteps (3K3K total tokens) is fed into a causally masked transformer. Standard causal self-attention over tokens with representations xix_i computes output hidden states ziz_i as: zi=∑j=1isoftmax({⟨qi,kj′⟩dk}j′=1i)jvjz_i = \sum_{j=1}^i \mathrm{softmax}\left( \left\{ \frac{\langle q_i, k_{j'} \rangle}{\sqrt{d_k}} \right\}_{j'=1}^i \right)_j v_j where qi,kj,vjq_i, k_j, v_j are linear query, key, and value projections. A linear prediction head placed on top of the hidden state corresponding to token sts_t autoregressively predicts action ata_t.

  2. Knowl 2 — Decision Transformer Training and Autoregressive Rollout Procedure

    algorithm

    Decision Transformer optimizes a sequence prediction loss over offline trajectory data and autoregressively generates actions at test time conditioned on target performance.

    Input: Offline dataset of trajectories D, context length K, training epochs E
    Output: Trained Decision Transformer model parameters theta
    Procedure Train(D, K, theta):
        for each epoch in 1 to E do
            for each sampled minibatch of sequences (R_t:t+K-1, s_t:t+K-1, a_t:t+K-1) from D do
                Compute token embeddings for R, s, a with learned timestep positional embeddings
                Pass interleaved tokens (R_1, s_1, a_1, ..., R_K, s_K) through causal Transformer
                Extract action predictions a_hat_t:t+K-1 from hidden states corresponding to s_t:t+K-1
                Compute action loss:
                    For continuous actions: L = mean((a_hat - a)^2)
                    For discrete actions: L = CrossEntropy(a_hat, a)
                Update theta via gradient descent on L
            end for
        end for
        return theta
    Procedure Rollout(theta, target_return, initial_state, K):
        R_seq <- [target_return]
        s_seq <- [initial_state]
        a_seq <- []
        t_seq <- [1]
        while episode is not done do
            Extract last K timesteps of sequences R_seq, s_seq, a_seq, t_seq
            Predict action a_t from DecisionTransformer(R_seq, s_seq, a_seq, t_seq)
            Execute a_t in environment; observe next state s_next and reward r_t
            R_next <- R_seq[end] - r_t
            Append R_next to R_seq, s_next to s_seq, a_t to a_seq, and length(R_seq) to t_seq
        end while

    During training, minibatches of length KK timesteps (3K3K tokens) are sampled from offline datasets. At evaluation time, the model is initialized with an initial state s1s_1 and a desired target return R^1\hat{R}_1. The predicted action ata_t is executed in the environment, and the return-to-go for the subsequent step is decremented by the received reward: R^t+1=R^t−rt\hat{R}_{t+1} = \hat{R}_t - r_t.

  3. Knowl 3 — Offline Continuous Control Performance on D4RL Benchmarks

    data/table

    Decision Transformer was evaluated across continuous control environments from the D4RL benchmark (HalfCheetah, Hopper, Walker2d) along with a 2D Reacher environment. The datasets encompass Medium-Expert (1M medium transitions concatenated with 1M expert transitions), Medium (1M transitions generated by a medium-level policy), and Medium-Replay (the replay buffer from training a policy to medium quality). Scores are normalized so that 100 represents expert policy return and 0 represents random policy return.

    Dataset Environment DT (Ours) CQL BEAR BRAC-v AWR BC
    Medium-Expert HalfCheetah 86.8 1.3 62.4 53.4 41.9 52.7 59.9
    Medium-Expert Hopper 107.6 1.8 111.0 96.3 0.8 27.1 79.6
    Medium-Expert Walker 108.1 0.2 98.7 40.1 81.6 53.8 36.6
    Medium-Expert Reacher 89.1 1.3 30.6 - - - 73.3
    Medium HalfCheetah 42.6 0.1 44.4 41.7 46.3 37.4 43.1
    Medium Hopper 67.6 1.0 58.0 52.1 31.1 35.9 63.9
    Medium Walker 74.0 1.4 79.2 59.1 81.1 17.4 77.3
    Medium Reacher 51.2 3.4 26.0 - - - 48.9
    Medium-Replay HalfCheetah 36.6 0.8 46.2 38.6 47.7 40.3 4.3
    Medium-Replay Hopper 82.7 7.0 48.6 33.7 0.6 28.4 27.6
    Medium-Replay Walker 66.6 3.0 26.7 19.2 0.9 15.5 36.9
    Medium-Replay Reacher 18.0 2.4 19.0 - - - 5.4
    Average (Without Reacher) 74.7 63.9 48.2 36.9 34.3 46.4
    Average (All Settings) 69.2 54.2 - - - 47.7

    Decision Transformer achieves higher average performance across the standard locomotion benchmarks (74.7 normalized score) than Conservative Q-Learning (CQL, 63.9), BEAR (48.2), BRAC-v (36.9), AWR (34.3), and Behavior Cloning (BC, 46.4), matching or exceeding baseline models across most subsets without dynamic programming or explicit value estimation.

  4. Knowl 4 — Offline Discrete Control Performance on Atari Benchmarks

    data/table

    Decision Transformer was evaluated on four Atari games (Breakout, Qbert, Pong, Seaquest) using a low-data regime comprising 1% of the DQN replay buffer (500k observed transitions out of 50 million). Scores are gamer-normalized, where 100 corresponds to professional human performance and 0 corresponds to a random policy.

    Game DT (Ours) CQL QR-DQN REM BC
    Breakout 267.5 97.5 211.1 17.1 8.9 138.9 61.7
    Qbert 15.4 11.4 104.2 0.0 0.0 17.3 14.7
    Pong 106.1 8.1 111.9 18.0 0.5 85.2 20.0
    Seaquest 2.5 0.4 1.7 0.4 0.7 2.1 0.3

    Using a context length of K=30K=30 (K=50K=50 for Pong), Decision Transformer matches or exceeds CQL in 3 of 4 games and outperforms QR-DQN, REM, and Behavior Cloning across the benchmark.

  5. Knowl 5 — Long-Term Credit Assignment in the Key-to-Door Environment

    empirical result

    In the Key-to-Door environment, an agent navigates three sequential rooms: (1) a room with a key, (2) an empty distractor room, and (3) a room with a door. A binary reward (+1+1) is granted only upon reaching the door in phase 3 if the key was picked up in phase 1. The task tests whether credit can be propagated across long horizons over intermediate distractor steps without learning a critic or Bellman updates.

    Models were trained exclusively on datasets generated by random actions, using the full episode length as context for Decision Transformer.

    Dataset DT (Ours) CQL BC %BC Random
    1K Random Trajectories 71.8% 13.1% 1.4% 69.9% 3.1%
    10K Random Trajectories 94.6% 13.3% 1.6% 95.1% 3.1%

    Temporal difference learning (CQL) fails to propagate value functions across the distractor room (13.1%13.1\% and 13.3%13.3\% success). Decision Transformer achieves 71.8%71.8\% (1k trajectories) and 94.6%94.6\% (10k trajectories), matching hindsight conditioning behavior cloning (%BC trained on success episodes). Analysis of attention weights shows that the self-attention mechanism directly links door-reaching tokens to key-pickup tokens, performing direct credit assignment.

  6. Knowl 6 — Decision Transformer Robustness to Delayed and Sparse Rewards

    empirical result

    To evaluate resilience to sparse reward signals, continuous control benchmarks on Hopper were modified to a delayed reward setting where the agent receives zero reward at intermediate timesteps and the full cumulative return ∑t=1Trt\sum_{t=1}^T r_t at the final timestep TT.

    Delayed (Sparse) Agnostic Original (Dense)
    Dataset Environment DT (Ours) CQL BC %BC DT (Ours) CQL
    Medium-Expert Hopper 107.3 3.5 9.0 59.9 102.6 107.6 111.0
    Medium Hopper 60.7 4.5 5.2 63.9 65.9 67.6 58.0
    Medium-Replay Hopper 78.5 3.7 2.0 27.6 70.6 82.7 48.6

    While temporal difference learning with value conservatism (CQL) collapses under delayed rewards (performance dropping to 9.09.0, 5.25.2, and 2.02.0), Decision Transformer performance remains virtually unaffected (107.3107.3, 60.760.7, and 78.578.5) because trajectory returns-to-go remain identical at t=1t=1, and self-attention models temporal dependencies directly without relying on dense per-step Bellman backups.

  7. Knowl 7 — Return-Target Controllability and Return Extrapolation

    empirical result

    When evaluated across continuous control (HalfCheetah, Hopper, Walker, Reacher) and discrete control (Breakout, Qbert, Pong, Seaquest), the average return accumulated during evaluation exhibits a near-linear correlation with the specified conditioning target R^1\hat{R}_1.

    In environments such as Pong, HalfCheetah, and Walker, the realized evaluation returns closely follow the oracle identity line across the full range of queried returns. Furthermore, on certain environments like Seaquest, conditioning Decision Transformer on return prompts higher than the maximum trajectory return present in the training dataset results in higher performance than the best trajectory in the dataset, demonstrating capacity for out-of-distribution return extrapolation.

  8. Knowl 8 — Impact of Context Window Length on Decision Transformer Performance

    empirical result

    Ablating the context length KK demonstrates that multi-step trajectory history is critical for Decision Transformer, even in environments where individual frames are stacked.

    Game DT (Context K=30K=30 or 5050) DT with no context (K=1K=1)
    Breakout 267.5 97.5 73.9 10
    Qbert 15.1 11.4 13.6 11.3
    Pong 106.1 8.1 2.5 0.2
    Seaquest 2.5 0.4 0.6 0.1

    Performance degrades substantially across all evaluated Atari games when context is restricted to K=1K=1. Conditioning on historical states, actions, and returns allows the transformer to identify the underlying policy distribution that generated previous transitions in mixed-quality offline data, aiding action selection and training dynamics.

  9. Knowl 9 — Percentile Behavior Cloning

    model/method

    Percentile Behavior Cloning (%BC) is an imitation learning baseline where standard supervised behavior cloning is trained exclusively on the top X%X\% of timesteps in an offline dataset, sorted by total trajectory return. Setting X=100%X = 100\% corresponds to standard behavior cloning on the full dataset, while X→0%X \to 0\% clones only the single highest-performing trajectory.

    %BC balances data efficiency against policy quality: smaller XX filters out suboptimal demonstrations but reduces sample size, harming generalization. In data-rich environments (such as D4RL), %BC at optimal XX matches or outperforms model-free offline RL algorithms (achieving up to 56.756.7 average score across D4RL subsets). However, choosing the optimal XX requires environment rollout evaluations, whereas Decision Transformer trains on 100%100\% of trajectories and conditions on target returns at test time, matching or exceeding optimal %BC without hyperparameter sweeps over dataset subsets and significantly outperforming %BC in low-data regimes (e.g., 1% Atari replay).

  10. Knowl 10 — Dynamic-Programming-Free Policy Improvement via Subpath Stitching on Graphs

    empirical result

    On a directed graph navigation MDP (20 nodes, edge sparsity coefficient 0.1, reward 0 at goal and -1 per step elsewhere), Decision Transformer was trained on 1,000 random walk trajectories of length T=10T=10 with no expert demonstrations. At test time, actions and return tokens were sampled using a generation prior favoring shorter path lengths: Pprior(R^=k)∝T+1−kP_{\mathrm{prior}}(\hat{R}=k) \propto T + 1 - k, modifying the return logits via: P(R^t∣s0:t,a0:t−1,R^0:t−1)∝PGPT(R^t∣s0:t,a0:t−1,R^0:t−1)×(Pprior(R^t))10P(\hat{R}_t \mid s_{0:t}, a_{0:t-1}, \hat{R}_{0:t-1}) \propto P_{\mathrm{GPT}}(\hat{R}_t \mid s_{0:t}, a_{0:t-1}, \hat{R}_{0:t-1}) \times \left( P_{\mathrm{prior}}(\hat{R}_t) \right)^{10}

    The model generates near-optimal shortest paths to the goal node. In 15.8%15.8\% of generated paths, the resulting optimal trajectory was not present as a contiguous sequence anywhere in the training data, demonstrating that sequence modeling combined with hindsight return conditioning can stitch together suboptimal segments to produce policy improvement without dynamic programming.

Coverage note — None was omitted; all primary architectural components, algorithmic workflows, benchmark evaluations (D4RL continuous control, Atari discrete control, Key-to-Door credit assignment), sparse reward studies, context length ablations, return controllability analyses, and graph illustrative experiments were captured.

References

  1. 1.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
  2. 2.Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
  3. 3.Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092, 2021.
  4. 4.Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International Conference on Machine Learning, 2020.
  5. 5.Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Deep reinforcement learning with relational inductive biases. In International Conference on Learning Representations, 2018.
  6. 6.Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, 2018.
  7. 7.Chia-Chun Hung, Timothy Lillicrap, Josh Abramson, Yan Wu, Mehdi Mirza, Federico Carnevale, Arun Ahuja, and Greg Wayne. Optimizing agent behavior over long time scales by transporting value. Nature communications, 10(1):1–12, 2019.
  8. 8.Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
  9. 9.Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
  10. 10.Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, 2013.
  11. 11.Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
  12. 12.Thomas Mesnard, Théophane Weber, Fabio Viola, Shantanu Thakoor, Alaa Saade, Anna Harutyunyan, Will Dabney, Tom Stepleton, Nicolas Heess, Arthur Guez, et al. Counterfactual credit assignment in model-free reinforcement learning. arXiv preprint arXiv:2011.09464, 2020.
  13. 13.Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, 2020.
  14. 14.Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
  15. 15.Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  16. 16.Richard S. Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In ICML, 1990.
  17. 17.Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, pages 12498–12509, 2019.
  18. 18.Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv:1906.00949, 2019.
  19. 19.Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019.
  20. 20.Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015.
  21. 21.Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020.
  22. 22.Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Conference on Artificial Intelligence, 2018.
  23. 23.Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
  24. 24.Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
  25. 25.David Raposo, Sam Ritter, Adam Santoro, Greg Wayne, Theophane Weber, Matt Botvinick, Hado van Hasselt, and Francis Song. Synthetic returns for long-term credit assignment. arXiv preprint arXiv:2102.12425, 2021.
  26. 26.Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning from imperfect demonstrations. arXiv preprint arXiv:1802.05313, 2018.
  27. 27.Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
  28. 28.Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995, 2019.
  29. 29.Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, 2019.
  30. 30.Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, 2019.
  31. 31.Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. In International Conference on Learning Representations, 2020.
  32. 32.Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
  33. 33.Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Advances in Neural Information Processing Systems, 2020.
  34. 34.Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010.13611, 2020.
  35. 35.Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giro-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, 2020.
  36. 36.Karl Pertsch, Youngwoon Lee, and Joseph J Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010.11944, 2020.
  37. 37.Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In International Conference on Learning Representations, 2021.
  38. 38.Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019.
  39. 39.Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Reset-free lifelong learning with skill-space planning. arXiv preprint arXiv:2012.03548, 2020.
  40. 40.Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, 2020.
  41. 41.Christopher Watkins. Learning from delayed rewards. 01 1989.
  42. 42.Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
  43. 43.Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojciech Jaśkowski, and Jürgen Schmidhuber. Training agents using upside-down reinforcement learning. arXiv preprint arXiv:1912.02877, 2019.
  44. 44.Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-conditioned policies. arXiv preprint arXiv:1912.13465, 2019.
  45. 45.Acting without rewards. 2019. URL https://ogma.ai/2019/08/acting-without-rewards/.
  46. 46.Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691–1703. PMLR, 2020.
  47. 47.Dibya Ghosh, Abhishek Gupta, Justin Fu, Ashwin Reddy, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals without reinforcement learning. arXiv preprint arXiv:1912.06088, 2019.
  48. 48.Keiran Paster, Sheila A McIlraith, and Jimmy Ba. Planning from pixels using inverse dynamics models. arXiv preprint arXiv:2012.02419, 2020.
  49. 49.Michael Janner, Qiyang Li, and Sergey Levine. Reinforcement learning as one big sequence modeling problem. arXiv preprint arXiv:2106.02039, 2021.
  50. 50.Johan Ferret, Raphaël Marinier, Matthieu Geist, and Olivier Pietquin. Self-attentional credit assignment for transfer in reinforcement learning. arXiv preprint arXiv:1907.08027, 2019.
  51. 51.Anna Harutyunyan, Will Dabney, Thomas Mesnard, Mohammad Azar, Bilal Piot, Nicolas Heess, Hado van Hasselt, Greg Wayne, Satinder Singh, Doina Precup, et al. Hindsight credit assignment. arXiv preprint arXiv:1912.02503, 2019.
  52. 52.Jose A Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. arXiv preprint arXiv:1806.07857, 2018.
  53. 53.Yang Liu, Yunan Luo, Yuanyi Zhong, Xi Chen, Qiang Liu, and Jian Peng. Sequence modeling of temporal credit assignment for episodic reinforcement learning. arXiv preprint arXiv:1905.13420, 2019.
  54. 54.Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Conference on Computer Vision and Pattern Recognition, 2019.
  55. 55.Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. Hafez: an interactive poetry generation system. In Proceedings of ACL, System Demonstrations, 2017.
  56. 56.Lilian Weng. Controllable neural text generation. lilianweng.github.io/lil-log, 2021. URL https://lilianweng.github.io/lil-log/2021/01/02/controllable-neural-text-generation.html.
  57. 57.Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects in neural language generation. arXiv preprint arXiv:1707.02633, 2017.
  58. 58.Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. Toward controlled generation of text. In International Conference on Machine Learning, 2017.
  59. 59.Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906.02361, 2019.
  60. 60.Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI conference on artificial intelligence, 2017.
  61. 61.Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.
  62. 62.Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858, 2019.
  63. 63.Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv:1912.02164, 2019.
  64. 64.Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. arXiv preprint arXiv:1805.06087, 2018.
  65. 65.Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.06367, 2020.
  66. 66.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  67. 67.Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, 2020.
  68. 68.Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  69. 69.Sam Ritter, Ryan Faulkner, Laurent Sartran, Adam Santoro, Matt Botvinick, and David Raposo. Rapid task-solving in novel environments. arXiv preprint arXiv:2006.03662, 2020.
  70. 70.Sudeep Dasari and Abhinav Gupta. Transformers for one-shot visual imitation. arXiv preprint arXiv:2011.05970, 2020.
  71. 71.Josh Abramson, Arun Ahuja, Iain Barr, Arthur Brussee, Federico Carnevale, Mary Cassin, Rachita Chhaparia, Stephen Clark, Bogdan Damoc, Andrew Dudzik, et al. Imitating interactive intelligence. arXiv preprint arXiv:2012.05672, 2020.
  72. 72.Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. Transformers: State-of-the-art natural language processing. In Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020.
  73. 73.Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.

Citation

MLA
Chen, L., et al. “Decision Transformer: Reinforcement Learning via Sequence Modeling”. arXiv, 2021, https://doi.org/10.48550/arxiv.2106.01345.
APA
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., & Mordatch, I. (2021). Decision Transformer: Reinforcement Learning via Sequence Modeling. arXiv. https://doi.org/10.48550/arxiv.2106.01345
Chicago
Chen, L., K. Lu, A. Rajeswaran, et al. 2021. “Decision Transformer: Reinforcement Learning via Sequence Modeling”. Preprint, ArXiv. https://doi.org/10.48550/arxiv.2106.01345.
Harvard
Chen, L. et al. (2021) “Decision Transformer: Reinforcement Learning via Sequence Modeling”. arXiv. Available at: https://doi.org/10.48550/arxiv.2106.01345.
Vancouver
1. Chen L, Lu K, Rajeswaran A, Lee K, Grover A, Laskin M, Abbeel P, Srinivas A, Mordatch I (2021) Decision Transformer: Reinforcement Learning via Sequence Modeling. https://doi.org/10.48550/arxiv.2106.01345

BibTeX

@misc{https://doi.org/10.48550/arxiv.2106.01345,
  doi = {10.48550/ARXIV.2106.01345},
  url = {https://arxiv.org/abs/2106.01345},
  author = {Chen, Lili and Lu, Kevin and Rajeswaran, Aravind and Lee, Kimin and Grover, Aditya and Laskin, Michael and Abbeel, Pieter and Srinivas, Aravind and Mordatch, Igor},
  keywords = {Machine Learning (cs.LG), Artificial Intelligence (cs.AI), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {Decision Transformer: Reinforcement Learning via Sequence Modeling},
  publisher = {arXiv},
  year = {2021},
  copyright = {arXiv.org perpetual, non-exclusive license}
}
Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Published with permission