Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

single GPU

A single GPU refers to a computing hardware setup or execution environment that relies on only one graphics processing unit to perform computational workloads. In parallel computing and machine learning, this architecture contrasts with multi-GPU configurations and distributed computing clusters, constraining all specialized parallel processing and high-bandwidth video memory to the physical limits of that solitary accelerator. Because high-demand tasks such as deep learning model training or inference often exceed the onboard memory capacity of an individual processor, single-GPU execution frequently requires specialized software techniques, including memory offloading, compression, and precision reduction, to efficiently process large datasets and models on a single device.

1 item

FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU

FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU

Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang

OrganizationsCarnegie Mellon UniversityETH ZurichHigher School of EconomicsMetaStanford UniversityUniversity of California BerkeleyYandex

Why you should read this

Presents FlexGen, an offloading engine that combines linear-programming-based tensor scheduling across GPU, CPU, and disk with 4-bit compression to achieve up to 100-fold higher throughput for 175B-parameter model inference on a single commodity GPU.

The high computational and memory requirements of large language model (LLM) inference make it feasible only with multiple high-end accelerators. Motivated by the emerging demand for latency-insensitive tasks with batched processing, this paper initiates the study of high-throughput LLM inference using limited resources, such as a single commodity GPU. We present FlexGen, a high-throughput generation engine for running LLMs with limited GPU memory. FlexGen can be flexibly configured under various hardware resource constraints by aggregating memory and computation from the GPU, CPU, and disk. By solving a linear programming problem, it searches for efficient patterns to store and access tensors. FlexGen further compresses the weights and the attention cache to 4 bits with negligible accuracy loss. These techniques enable FlexGen to have a larger space of batch size choices and thus significantly increase maximum throughput. As a result, when running OPT-175B on a single 16GB GPU, FlexGen achieves significantly higher throughput compared to state-of-the-art offloading systems, reaching a generation throughput of 1 token/s for the first time with an effective batch size of 144. On the HELM benchmark, FlexGen can benchmark a 30B model with a 16GB GPU on 7 representative sub-scenarios in 21 hours. The code is available at https://github.com/FMInference/FlexGen.

Added

2026-09-26