Back

Efficient Streaming Language Models with Attention Sinks

Guangxuan XiaoYuandong TianBeidi ChenSong HanMike Lewis

article2023ICLR2,442 citations

Introduces StreamingLLM, an efficient framework that exploits the attention sink phenomenon to enable pre-trained large language models to generalize to infinite sequence lengths without fine-tuning while achieving up to a 22.2x speedup over sliding-window recomputation.

Listen

Deploying large language models in continuous, streaming environments like multi-round chatbots faces significant technical hurdles. Standard architectures cache all past tokens during generation, which leads to excessive memory consumption and latency. Meanwhile, standard models degrade sharply when conversations exceed their fixed training window, and basic sliding-window approaches collapse completely once the very first tokens are discarded.

The article demonstrates why standard sliding-window attention fails and evaluates whether large language models can handle infinite input streams without fine-tuning or performance degradation.

The researchers analyzed attention score distributions across multiple model families, including Llama-2, MPT, Falcon, and Pythia. They discovered that transformer attention mechanisms disproportionately concentrate high numerical scores on the initial sequence tokens, creating an effect termed attention sinks. Building on this discovery, the authors evaluated StreamingLLM, an inference framework combining initial sink tokens with a rolling cache of recent tokens, across synthetic benchmarks, question-answering tasks, and text sequences reaching 4 million tokens.

The analysis produced several critical findings. First, retaining as few as four initial sink tokens alongside recent context fully restores model stability without fine-tuning, whereas discarding them causes severe performance collapse. Second, StreamingLLM enabled stable language modeling on sequences exceeding 4 million tokens across four major model families. Third, the framework delivered up to a 22.2-fold decoding speedup compared to sliding window recomputation baselines while maintaining identical memory usage. Finally, pre-training models from scratch with a single designated placeholder token completely eliminated the need to retain multiple initial tokens during streaming inference.

These findings indicate that existing language models can be deployed in persistent, low-latency streaming applications without costly retraining or architectural redesigns. Decoupling sequence length from pre-training window limits directly reduces infrastructure costs and enables reliable 24/7 conversational agents.

Organizations deploying streaming language models should adopt attention-sink caching strategies to improve throughput and reduce memory overhead. Additionally, teams pre-training new base models should incorporate a dedicated placeholder token at the start of training sequences to optimize downstream streaming performance.

A key limitation is that StreamingLLM preserves stability over recent contexts but does not extend long-term memory. Tasks requiring comprehensive retrieval over distant historical inputs still require specialized external memory or retrieval systems. The reported results are backed by high empirical confidence across multiple model scales and standard benchmarks.

Carnegie Mellon UniversityMassachusetts Institute of TechnologyMetaNVIDIAcachecache sizecachingAttentionTokensModels
arXiv: 2309.17453mit-han-lab/streaming-llm
Cover for Efficient Streaming Language Models with Attention Sinks

Abstract

Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach -- but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attention sink, that keeping the KV of initial tokens will largely recover the performance of window attention. In this paper, we first demonstrate that the emergence of attention sink is due to the strong attention scores towards initial tokens as a "sink" even if they are not semantically important. Based on the above analysis, we introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence lengths without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more. In addition, we discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment. In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup. Code and datasets are provided at this https URL.

Topics

cachecache sizecachingdialoguelanguage modellarge language modelssink nodesstreaming applicationstraining sequencessliding windows

Keywords

AttentionTokensModelsTokenInitial TokensSinkLanguageStreamingLLMLanguage ModelsAttention Sinks

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 StreamingLLM
  • 3.1 The Failure of Window Attention and Attention Sinks
  • 3.2 Rolling KV Cache with Attention Sinks
  • 3.3 Pre-Training LLMs with Attention Sinks
  • 4 Experiments
  • 4.1 Language Modeling on Long Texts Across LLM Families and Scales
  • 4.2 Results of Pre-Training with a Sink Token
  • 4.3 Results on Streaming Question Answering with Instruction-tuned Models
  • 4.4 Ablation Studies
  • 4.5 Efficency Results
  • 5 Conclusion
  • References
  • A Discussions
  • B Additional Related Works
  • C Accuracy on StreamEval with Increasing Query-Answer Line Distance
  • D Long-Range Benchmark Evaluation
  • E Llama-2-7B Attention Visualization on Longer Sequences
  • F Quatitative Analysis of Attention Sinks in Long Inputs
  • G Llama-2-70B Attention Visualization
  • H Attention Sinks in Encoder Transformers
  • I Using More Sink Tokens in the Pre-Training Stage

Knowls

  1. Knowl 1 — Attention Sink Phenomenon in Autoregressive Transformers

    definition

    In autoregressive Transformer language models, an attention sink refers to initial tokens in a sequence that receive a disproportionately high share of attention score mass across layers and attention heads, regardless of their semantic relevance to the context.

    This behavior arises from the formulation of the SoftMax operation in multi-head self-attention:

    SoftMax(x)i=exi∑j=1Nexj\text{SoftMax}(x)_i = \frac{e^{x_i}}{\sum_{j=1}^N e^{x_j}}

    where x∈RNx \in \mathbb{R}^N represents the scaled dot-product query-key logits for NN contextual tokens. Because the SoftMax denominator requires all attention probabilities across visible tokens to sum to 1, an attention head must allocate any unneeded attention score mass somewhere, even when the current query has no strong semantic affinity with any preceding context token. Because initial tokens are causally visible to every subsequent token in autoregressive modeling, the network naturally learns to dump unneeded attention mass onto these starting tokens. Evicting initial tokens removes a major component of the SoftMax denominator, distorting attention score distributions across the entire network and causing immediate perplexity explosion.

  2. Knowl 2 — StreamingLLM Rolling KV Cache Framework

    model/method

    StreamingLLM is an inference framework that allows autoregressive large language models trained with finite attention windows to decode over arbitrarily long sequences without fine-tuning, maintaining O(1)O(1) memory usage and O(TL)O(T L) total time complexity for generating TT tokens with cache capacity LL.

    The Key-Value (KV) cache in StreamingLLM is partitioned into two distinct segments:

    1. Attention Sinks: A small, fixed set of initial tokens (typically 4 initial tokens) whose Key and Value states are retained permanently in the cache to stabilize the SoftMax attention denominator.
    2. Rolling KV Cache: A sliding window that dynamically retains the L−kL - k most recent tokens (where LL is total cache size and kk is the number of attention sinks), preserving local conversational context for fluent language generation.

    When new tokens are decoded beyond the cache capacity LL, the oldest tokens within the rolling portion are evicted while the initial sink tokens are strictly preserved.

  3. Knowl 3 — Cache-Centric Positional Encoding Assignment

    model/method

    To preserve relative positional encoding coherence when tokens are evicted from the KV cache, StreamingLLM assigns positional indices based on a token's index within the active cache rather than its original absolute position in the input text stream. For instance, if the cache holds initial tokens [0,1,2,3][0, 1, 2, 3] and recent tokens [6,7,8][6, 7, 8], the positions mapped to these tokens during the decoding of token 9 are assigned contiguously as [0,1,2,3,4,5,6,7][0, 1, 2, 3, 4, 5, 6, 7] rather than [0,1,2,3,6,7,8][0, 1, 2, 3, 6, 7, 8].

    Implementation depends on the positional encoding mechanism:

    • Rotary Position Embeddings (RoPE): Key vectors are stored in the KV cache in their unrotated state (prior to applying RoPE). At each decoding step, rotary transformations corresponding to active cache positions are applied dynamically to the keys in the cache.
    • Attention with Linear Biases (ALiBi): Contiguous linear distance biases corresponding to relative cache distances are applied directly to the query-key attention scores, avoiding discontinuous step changes.

  4. Knowl 4 — Pre-Training LLMs with a Dedicated Sink Token

    model/method

    Standard autoregressive LLMs distribute attention sink weights across multiple initial tokens because arbitrary text chunks start with random tokens during pre-training. Pre-training models with a single dedicated learnable placeholder token ("Sink Token") prepended to all training sequences concentrates the attention sink entirely onto that dedicated token.

    At inference time, a model pre-trained with a sink token requires caching only that single sink token alongside recent tokens (1+(L−1)1 + (L-1) cache configuration) to achieve full streaming stability, eliminating the need to preserve multiple natural language starting tokens.

    An alternative approach tested is SoftMax-off-by-One (Zero Sink), which modifies the attention normalization function to:

    SoftMax1(x)i=exi1+∑j=1Nexj\text{SoftMax}_1(x)_i = \frac{e^{x_i}}{1 + \sum_{j=1}^N e^{x_j}}

    where xix_i is the query-key logit for token ii. This is functionally equivalent to prepending an all-zero Key and Value vector, allowing contextual attention weights to sum to less than one.

  5. Knowl 5 — Semantic Independence of Attention Sinks

    empirical result

    The attention sink effect is determined by token position in the autoregressive visibility hierarchy rather than token semantics. When the first four tokens of an input sequence are replaced entirely with linebreak characters (\n), language modeling perplexity is restored to the same level as retaining the original text tokens.

    Cache Configuration Perplexity (↓\downarrow)
    0+10240 + 1024 (Window Attention) 5158.07
    4+10204 + 1020 (Original 4 Initial Tokens + Recent 1020) 5.40
    4"\n"+10204\text{"\textbackslash n"} + 1020 (4 Linebreak Tokens + Recent 1020) 5.60

    Perplexities were measured on the first book (65K tokens) of the PG-19 test set using Llama-2-13B. Furthermore, ablations show that retaining 1 or 2 initial tokens only partially restores perplexity (e.g., Llama-2-7B perplexity drops from 3359.95 with 0 sinks to 11.88 with 1 sink and 10.51 with 2 sinks on 400K PG-19 tokens), whereas retaining 4 initial sink tokens reaches 9.59, beyond which additional sink tokens yield diminishing returns.

  6. Knowl 6 — Infinite-Length Language Modeling Perplexity

    empirical result

    StreamingLLM maintains stable language modeling perplexity across extended sequences exceeding 4 million tokens on the concatenated PG-19 benchmark (100 books) across multiple LLM families and model sizes, including Llama-2 (7B, 13B, 70B), Falcon (7B, 40B), Pythia (2.8B, 6.9B, 12B), and MPT (7B, 30B).

    While dense attention fails once sequence length exceeds the pre-training context window size, and standard sliding window attention fails immediately once initial tokens are evicted from the KV cache, StreamingLLM matches the perplexity trajectory of the sliding window with re-computation oracle baseline continuously across all 4 million tokens without degradation.

  7. Knowl 7 — Per-Token Decoding Latency and Memory Comparison

    empirical result

    StreamingLLM achieves a per-token decoding speedup of up to 22.2×22.2\times over the sliding window with re-computation baseline on an NVIDIA A6000 GPU while maintaining a constant memory footprint.

    At a cache size of 4096 tokens on Llama-2-7B, sliding window with re-computation requires 1411 ms per token due to quadratic O(TL2)O(T L^2) attention recomputations, whereas StreamingLLM requires 65 ms per token (O(TL)O(T L) total complexity). On Llama-2-13B at a cache size of 4096 tokens, re-computation latency is 2355 ms per token compared to 106 ms for StreamingLLM. Both methods maintain identical fixed GPU memory footprints (14 to 21 GB for 7B; 25 to 36 GB for 13B across cache sizes from 256 to 4096).

  8. Knowl 8 — StreamEval Streaming Question Answering Evaluation

    empirical result

    On the StreamEval streaming question-answering benchmark, text is streamed continuously with queries inserted every 10 lines, where the ground-truth target is located exactly 20 lines prior (460 tokens distance). StreamingLLM maintains constant accuracy across streaming inputs up to 120,000 tokens for Llama-2-7B-Chat, Llama-2-13B-Chat, LongChat-7b-v1.5-32k, and Llama-2-7B-32K-Instruct.

    In contrast, dense attention encounters Out-of-Memory (OOM) failures or collapses beyond the pre-training window length, and standard window attention drops to near-zero accuracy as soon as the cache size is exceeded (e.g., scoring 0.12%−3.58%0.12\% - 3.58\% accuracy on streaming concatenated ARC benchmarks compared to 71.34%−91.37%71.34\% - 91.37\% achieved by StreamingLLM, which matches sample-by-sample one-shot performance).

  9. Knowl 9 — Context Horizon and Long-Term Memory Limitation of StreamingLLM

    limitation

    StreamingLLM enables models to generate fluent text indefinitely over an open stream, but it does not expand the effective context window size or provide long-term associative memory. The model can only attend to information currently held in its rolling KV cache window (LL tokens).

    On StreamEval evaluations with varying query-answer token distances, model accuracy drops to 0% as soon as the query-answer distance exceeds the rolling cache capacity LL (e.g., with cache configuration 4+20444+2044, accuracy is 75.30%75.30\% at 1840 token distance but drops to 0.00%0.00\% at 2300 token distance). Similarly, on LongBench tasks requiring long-range dependencies across the entire sequence (such as NarrativeQA and HotpotQA), StreamingLLM with a standard 4+34964+3496 configuration underperforms truncation baselines unless the attention sink size is explicitly configured to retain all initial prompt context.

Coverage note — No substantial contributed material was omitted; all key theoretical observations, methodology details (including cache indexing and pre-training sink tokens), empirical benchmarks (PG-19, StreamEval, ARC), efficiency benchmarks, and limitations are represented.

References

  1. 1.Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. Etc: Encoding long and structured inputs in transformers, 2020.
  2. 2.Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B: an open large language model with state-of-the-art performance. 2023.
  3. 3.Sotiris Anagnostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurelien Lucchi, and Thomas Hofmann. Dynamic context pruning for efficient and interpretable autoregressive transformers, 2023.
  4. 4.Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023.
  5. 5.Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer, 2020. arXiv:2004.05150.
  6. 6.Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
  7. 7.Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
  8. 8.bloc97. NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation., 2023. URL https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/.
  9. 9.Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing, 2023.
  10. 10.Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  11. 11.Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
  12. 12.Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023. arXiv: 2306.15595.
  13. 13.Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/.
  14. 14.Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. 2019.
  15. 15.Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018.
  16. 16.Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. 2023.
  17. 17.Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness, 2022. arXiv:2205.14135.
  18. 18.Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers, 2023.
  19. 19.Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers, 2021.
  20. 20.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019. URL https://api.semanticscholar.org/CorpusID:52967399.
  21. 21.Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
  22. 22.Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
  23. 23.Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.
  24. 24.Tanya Goyal and Greg Durrett. Evaluating factuality in generation with dependency-level entailment. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online, 2020. Association for Computational Linguistics.
  25. 25.Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. LM-Infinite: Simple on-the-fly length generalization for large language models, 2023.
  26. 26.Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, December 2020.
  27. 27.Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization, 2021.
  28. 28.kaiokendev. Things I’m learning while training superhot., 2023. URL https://kaiokendev.github.io/til#extending-context-to-8k.
  29. 29.Ehsan Kamalloo, Nouha Dziri, Charles L. A. Clarke, and Davood Rafiei. Evaluating open-domain question answering in the era of large language models, 2023.
  30. 30.Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th International Conference on Learning Representations, ICLR 2020. OpenReview.net, April 2020.
  31. 31.Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge, 2017.
  32. 32.Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, , and Hao Zhang. How long can open-source llms truly promise on context length?, June 2023. URL https://lmsys.org/blog/2023-06-29-longchat.
  33. 33.Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, 2023.
  34. 34.Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018.
  35. 35.Evan Miller. Attention is off by one, 2023. URL https://www.evanmiller.org/attention-is-off-by-one.html.
  36. 36.OpenAI. Gpt-4 technical report, 2023.
  37. 37.Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1525–1534, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1144. URL https://aclanthology.org/P16-1144.
  38. 38.Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models, 2023.
  39. 39.Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. arXiv preprint arXiv:2211.05102, 2022.
  40. 40.Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=R8sQPpGCv0.
  41. 41.Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
  42. 42.Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2020.
  43. 43.Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Llama: Open foundation models for code, 2023.
  44. 44.Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641, 2019.
  45. 45.John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Felipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, et al. Chatgpt: Optimizing language models for dialogue. OpenAI blog, 2022.
  46. 46.Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021.
  47. 47.Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023.
  48. 48.Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys, 55(6), dec 2022. ISSN 0360-0300.
  49. 49.MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023. URL www.mosaicml.com/blog/mpt-7b. Accessed: 2023-05-05.
  50. 50.Together. Llama-2-7b-32k-instruct — and fine-tuning for llama-2 models with together api, June 2023. URL https://together.ai/blog/llama-2-7b-32k-instruct.
  51. 51.Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.
  52. 52.Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
  53. 53.Hanrui Wang, Zhekai Zhang, and Song Han. Spatten: Efficient sparse attention architecture with cascade token and head pruning. HPCA, 2021.
  54. 54.Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. 2020.
  55. 55.Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface’s transformers: State-of-the-art natural language processing, 2020.
  56. 56.Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, 2023.
  57. 57.Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
  58. 58.Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big Bird: Transformers for longer sequences. In Proc. of NeurIPS, volume 33, 2020a.
  59. 59.Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020. Curran Associates, Inc., 2020b.
  60. 60.Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? CoRR, abs/1905.07830, 2019. URL http://arxiv.org/abs/1905.07830.
  61. 61.Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
  62. 62.Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. Benchmarking large language models for news summarization, 2023a.
  63. 63.Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o: Heavy-hitter oracle for efficient generative inference of large language models, 2023b.

Citation

MLA
Xiao, G., et al. “Efficient Streaming Language Models with Attention Sinks”. arXiv, 2023, http://arxiv.org/abs/2309.17453v4.
APA
Xiao, G., Tian, Y., Chen, B., Han, S., & Lewis, M. (2023). Efficient Streaming Language Models with Attention Sinks. arXiv. http://arxiv.org/abs/2309.17453v4
Chicago
Xiao, G., Y. Tian, B. Chen, S. Han, and M. Lewis. 2023. “Efficient Streaming Language Models with Attention Sinks”. arXiv. http://arxiv.org/abs/2309.17453v4.
Harvard
Xiao, G. et al. (2023) “Efficient Streaming Language Models with Attention Sinks”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2309.17453v4.
Vancouver
1. Xiao G, Tian Y, Chen B, Han S, Lewis M (2023) Efficient Streaming Language Models with Attention Sinks. arXiv

BibTeX

@article{xiao2023efficient,
  title = {Efficient Streaming Language Models with Attention Sinks},
  author = {Xiao, Guangxuan and Tian, Yuandong and Chen, Beidi and Han, Song and Lewis, Mike},
  year = {2023},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2309.17453v4},
  eprint = {2309.17453}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/