Energy and Policy Considerations for Deep Learning in NLP

Emma StrubellAnanya GaneshAndrew McCallum

article2019ACL3,657 citations

Quantifies the massive carbon footprint and financial expense of training modern natural language processing models while offering concrete recommendations to prioritize computational efficiency and equitable access in AI research.

Listen

Recent progress in deep neural networks has produced large gains in accuracy on many natural language processing tasks, yet these gains now depend on training runs that consume substantial electricity and specialized hardware. The resulting financial expense and carbon emissions raise concerns about who can participate in cutting-edge research and about the environmental impact of widespread model development. This paper sets out to measure those costs for several widely used models and for the full process of developing one state-of-the-art system, then to offer practical steps the field can take in response.

The authors estimated energy use by running the models on representative GPUs, recording average power draw, and scaling the measurements by the training durations reported in the original publications. They converted kilowatt-hours to carbon dioxide emissions with U.S. Environmental Protection Agency factors and to dollar costs with current cloud-compute prices. A separate case study tallied every training job performed over six months while developing a multi-task parsing pipeline, producing nearly 4,800 separate runs.

Training one large Transformer model emitted roughly 192 pounds of CO2 and cost a few hundred dollars on the cloud; the much larger BERT model required the equivalent of a transcontinental flight in emissions and several thousand dollars. A full neural-architecture search multiplied those figures by more than three thousand. Developing the parsing pipeline over six months consumed the equivalent of 27 years of GPU time and an estimated $103,000 to$ 350,000 in cloud fees. These figures already exceed the lifetime emissions of an average car for some experiments and quickly become prohibitive for groups without large budgets.

The results show that accuracy improvements are being purchased at rapidly rising financial and environmental cost, concentrating progress in well-funded industry labs and leaving academic researchers at a growing disadvantage. They also indicate that the bulk of the expense lies not in a single final training run but in the repeated experiments needed to tune and validate new models.

The authors therefore recommend three concrete changes: papers should report training time and sensitivity to hyperparameters so that later users can judge true cost; funding agencies should create shared academic computing resources to restore equitable access; and both industry and academia should prioritize research on more efficient algorithms and hardware, including better hyperparameter-search methods that reduce the need for exhaustive trials. These steps would lower barriers without sacrificing scientific quality.

The estimates rely on average U.S. electricity sources and a global data-center efficiency factor; actual emissions vary by region and by whether providers use renewables. Hardware details for TPUs were unavailable, so some comparisons rest on partial data. Nonetheless, the measured orders of magnitude are consistent across models and provide a reliable basis for the policy conclusions.

College of Information and Computer Sciences University of Massachusetts Amherst computational resources computer hardware deep learning Energy Consumption Carbon Emission Transformer

arXiv: 1906.02243

Paper: Scaling Laws for Neural Language Models, Jared Kaplan et al. (2020). Understanding neural scaling laws is essential before evaluating the computational and carbon footprints of training ever-larger language models.
Paper: Language Models are Few-Shot Learners, T. B. Brown et al. (2020). This seminal description of GPT-3 establishes the massive scale and resource demands that directly motivate the source paper's environmental and cost audit.

Paper: On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜, Emily M. Bender et al. (2021). This paper expands significantly on the environmental and societal critiques introduced in the source by framing large language models as uncurated stochastic parrots.
Paper: GLaM: Efficient Scaling of Language Models with Mixture-of-Experts, Nan Du et al. (2022). This work directly addresses the energy and scaling challenges highlighted in the source by deploying mixture-of-experts architectures to reduce training costs.

Cover for Energy and Policy Considerations for Deep Learning in NLP

Abstract

Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data. These models have obtained notable gains in accuracy across many NLP tasks. However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption. As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware. In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP. Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.

1 Introduction
2 Methods
2.1 Models
3 Related work
4 Experimental results
4.1 Cost of training
4.2 Cost of development: Case study
5 Conclusions
Authors should report training time and sensitivity to hyperparameters.
Academic researchers need equitable access to computation resources.
Researchers should prioritize computationally efficient hardware and algorithms.
Acknowledgements
References

Knowls

Knowl 1 — Total Power Consumption Estimation Formula for Deep Learning Training
equation
The estimated total electrical power consumption $p_t$ (in kilowatt-hours, $\text{kWh}$ ) required to train a machine learning model over a duration of $t$ hours is given by:

$p_t = \frac{1.58 \cdot t \cdot (p_c + p_r + g \cdot p_g)}{1000}$

where:
- $t \in \mathbb{R}^+$ is the total training time in hours.
- $p_c \in \mathbb{R}^+$ is the average power draw in watts ( $\text{W}$ ) across all CPU sockets during training.
- $p_r \in \mathbb{R}^+$ is the average power draw in watts ( $\text{W}$ ) from all DRAM (main memory) sockets during training.
- $p_g \in \mathbb{R}^+$ is the average power draw of a single GPU in watts ( $\text{W}$ ) during training.
- $g \in \mathbb{N}^+$ is the number of GPUs simultaneously used for training.
- The coefficient $1.58$ is the Power Usage Effectiveness (PUE) multiplier, which accounts for the ancillary energy consumption of data center infrastructure (primarily cooling), based on the 2018 global average for data centers.
- The divisor $1000$ converts watt-hours ( $\text{Wh}$ ) to kilowatt-hours ( $\text{kWh}$ ).

Knowl 2 — Estimated Energy Consumption, Carbon Footprint, and Financial Cost of Training Standard NLP Models

data/table

The table below details the estimated hardware runtime, energy consumed, carbon dioxide equivalent emissions ( $\text{CO}_2\text{e}$ in pounds), and cloud computing costs required to train several standard natural language processing (NLP) models. GPU cost lower bounds are based on pre-emptible P100/V100 rates ($0.43--$0.74/hr) and upper bounds on on-demand rates ($1.46--$2.48/hr). TPU lower bounds reflect pre-emptible rates ($1.46--$2.40/hr) and upper bounds reflect on-demand rates ($4.50--$8.00/hr).

Model	Hardware	Power (W)	Hours	kWh $\cdot$ PUE	CO $_2$ e (lbs)	Cloud compute cost
T2T $_{\text{base}}$	P100 $\times$ 8	1415.78	12	27	26	$41–$140
T2T $_{\text{big}}$	P100 $\times$ 8	1515.43	84	201	192	$289–$981
ELMo	P100 $\times$ 3	517.66	336	275	262	$433–$1472
BERT $_{\text{base}}$	V100 $\times$ 64	12,041.51	79	1507	1438	$3751–$12,571
BERT $_{\text{base}}$	TPUv2 $\times$ 16	—	96	—	—	$2074–$6912
NAS	P100 $\times$ 8	1515.43	274,120	656,347	626,155	$942,973–$3,201,722
NAS	TPUv2 $\times$ 1	—	32,623	—	—	$44,055–$146,848
GPT-2	TPUv3 $\times$ 32	—	168	—	—	$12,902–$43,008

The results highlight that training standard transformer models produces substantial carbon emissions (e.g., GPU-trained $\text{BERT}_{\text{base}}$ produces roughly the emissions of a trans-American flight). Neural Architecture Search (NAS) requires orders of magnitude more compute, generating over $626{,}000\text{ lbs of }\text{CO}_2\text{e}$ on GPUs to attain a marginal accuracy improvement ( $+0.1\text{ BLEU}$ ). TPUs demonstrate greater cost efficiency than GPUs on compatible parallel workloads.

Knowl 3 — Carbon Dioxide Equivalent Emissions Conversion Formula for Power Consumption
equation
The estimated carbon dioxide equivalent emissions $\text{CO}_2\text{e}$ (measured in pounds) resulting from electrical energy consumption is calculated as:

$\text{CO}_2\text{e} = 0.954 \cdot p_t$

where:
- $p_t \in \mathbb{R}^+$ is the total power consumed in kilowatt-hours ( $\text{kWh}$ ).
- $0.954$ is the average conversion factor (in pounds of $\text{CO}_2\text{e}$ per $\text{kWh}$ ) based on the U.S. Environmental Protection Agency (EPA) 2018 eGRID database, reflecting the nationwide aggregate fuel generation mix (natural gas, coal, nuclear, and renewables) in the United States.

Knowl 4 — Computational and Financial Cost of NLP Model Research and Development: Case Study on LISA

data/table

A full research and development (R&D) case study tracking the development of the Linguistically-Informed Self-Attention (LISA) multi-task semantic role labeling model over a 172-day period (approximately 6 months) reveals the resource multiplier introduced by experimentation and hyperparameter tuning. The project executed 123 grid searches totaling 4,789 training jobs (ranging from 3 minutes to 9 days; average 52 hours per job) using NVIDIA Titan X (72%) and M40 (28%) GPUs, consuming 239,942 GPU hours (9,998 GPU days or ~27 GPU years).

While training a single final model instance is inexpensive ($52--$175 in cloud compute), hyperparameter tuning for a new dataset (estimated at 24 jobs) costs over $1,200--$4,200, and full R&D development incurs between $103,000 and $350,000 in cloud compute costs (or $9,870 in direct electricity at $0.12/kWh).

Knowl 5 — Comparative Carbon Footprint of NLP Model Training Against Baseline Human Activities

data/table

The table below contextualizes estimated $\text{CO}_2\text{e}$ emissions generated by training deep learning NLP models against common personal, transportation, and societal energy benchmarks:

Consumption	CO $_2$ e (lbs)
Air travel, 1 person, NY $\leftrightarrow$ SF	1984
Human life, avg, 1 year	11,023
American life, avg, 1 year	36,156
Car, avg incl. fuel, 1 lifetime	126,000
Training one model (GPU)
NLP pipeline (parsing, SRL)	39
w/ tuning experiments	78,468
Transformer (big)	192
w/ neural arch. search	626,155

A single training run of an NLP pipeline produces relatively modest emissions (39 lbs $\text{CO}_2\text{e}$ ), but comprehensive tuning and experimentation inflate emissions to 78,468 lbs (exceeding two years of average American per-capita emissions). Neural architecture search on a large Transformer model generates 626,155 lbs of $\text{CO}_2\text{e}$ , nearly five times the lifetime emissions of an average passenger automobile.

Knowl 6 — Hardware Power Measurement Protocol for Deep Learning Workloads
experimental setup
To empirically measure power consumption during deep neural network training:
1. GPU Power Consumption: The NVIDIA System Management Interface (nvidia-smi) is repeatedly polled during training execution to sample instantaneous GPU power draw in watts, and the mean power draw ( $p_g$ ) across all sampled intervals is computed.
2. CPU and Memory Power Consumption: Intel's Running Average Power Limit (RAPL) interface is queried to measure instantaneous power consumption across all CPU sockets ( $p_c$ ) and all DRAM sockets ( $p_r$ ).
3. Duration: Models are trained using their standard provided configurations on local hardware (e.g., single NVIDIA Titan X or clusters of GTX 1080 Ti GPUs) for up to 1 day to collect steady-state power draw profiles before scaling estimates to the full reported training durations.

Knowl 7 — Energy Generation Profiles of Major Cloud Computing Providers and Countries

data/table

The environmental impact of deep learning compute depends heavily on the primary energy sources powering data centers. The table below presents the percentage breakdown of energy sourced from renewables (hydro, solar, wind), natural gas, coal, and nuclear power across key geographic regions and major cloud providers:

Consumer	Renewable	Gas	Coal	Nuclear
China	22%	3%	65%	4%
Germany	40%	7%	38%	13%
United States	17%	35%	27%	19%
Amazon-AWS	17%	24%	30%	26%
Google	56%	14%	15%	10%
Microsoft	32%	23%	31%	10%

The fuel breakdown of Amazon Web Services closely resembles the average United States national electrical grid profile, supporting the use of the U.S. national average emissions factor ( $0.954\text{ lbs }\text{CO}_2\text{e}/\text{kWh}$ ) for cloud-based carbon estimations.

Knowl 8 — Policy and Practice Recommendations for Sustainable and Equitable NLP Research
model/method
To address rising environmental costs and inequitable access in deep learning NLP research, three actionable policies are recommended:
1. Standardized Compute Reporting: Authors proposing machine learning models intended for downstream fine-tuning or retraining should report hardware-independent training time metrics (such as gigaflops required to reach convergence) alongside sensitivity to hyperparameters (such as performance variance across the hyperparameter search space).
2. Public Academic Compute Infrastructure: Funding agencies (such as the U.S. National Science Foundation) should pool resources to establish shared academic compute clouds. Purchasing centralized on-premise hardware (e.g., 8-GPU servers at approximately $20,000 each) is roughly half the cost of equivalent on-demand cloud services and prevents cumulative spending from locking non-profit academic institutions out of large-scale deep learning research.
3. Integration of Efficient Hyperparameter Search: Machine learning software developers and framework maintainers (e.g., PyTorch, TensorFlow) should integrate easy-to-use APIs for efficient hyperparameter optimization algorithms (such as random search and Bayesian optimization) into standard model training workflows to eliminate reliance on computationally wasteful brute-force grid search.

Coverage note — No substantial contributed material was omitted; the knowls fully capture the power and carbon conversion equations, the hardware profiling experimental protocol, the empirical cost and carbon tables across NLP architectures, the LISA full R&D life-cycle case study, and the actionable policy and workflow recommendations.

References

1.Rhonda Ascierto. 2018. Uptime Institute Global Data Center Survey. Technical report, Uptime Institute.
2.Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In 3rd International Conference for Learning Representations (ICLR), San Diego, California, USA.
3.James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb):281–305.
4.James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization. In Advances in neural information processing systems, pages 2546–2554.
5.Bruno Burger. 2019. Net Public Electricity Generation in Germany in 2018. Technical report, Fraunhofer Institute for Solar Energy Systems ISE.
6.Alfredo Canziani, Adam Paszke, and Eugenio Culurciello. 2016. An analysis of deep neural network models for practical applications.
7.Gary Cook, Jude Lee, Tamina Tsai, Ada Kongn, John Deans, Brian Johnson, Elizabeth Jardim, and Brian Johnson. 2017. Clicking Clean: Who is winning the race to build a green internet? Technical report, Greenpeace.
8.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL.
9.Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In ICLR.
10.EPA. 2018. Emissions & Generation Resource Integrated Database (eGRID). Technical report, U.S. Environmental Protection Agency.
11.Christopher Forster, Thor Johnsen, Swetha Mandava, Sharath Turuvekere Sreenivas, Deyu Fu, Julie Bernauer, Allison Gray, Sharan Chetlur, and Raul Puri. 2019. BERT Meets GPUs. Technical report, NVIDIA AI.
12.Da Li, Xinbo Chen, Michela Becchi, and Ziliang Zong. 2016. Evaluating the energy efficiency of deep convolutional neural networks on cpus and gpus. 2016 IEEE International Conferences on Big Data and Cloud Computing (BDCloud), Social Computing and Networking (SocialCom), Sustainable Computing and Communications (SustainCom) (BDCloud-SocialCom-SustainCom), pages 477–484.
13.Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421. Association for Computational Linguistics.
14.Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL.
15.Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
16.Jasper Snoek, Hugo Larochelle, and Ryan P Adams. 2012. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pages 2951–2959.
17.David R. So, Chen Liang, and Quoc V. Le. 2019. The evolved transformer. In Proceedings of the 36th International Conference on Machine Learning (ICML).
18.Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-Informed Self-Attention for Semantic Role Labeling. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Brussels, Belgium.
19.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In 31st Conference on Neural Information Processing Systems (NIPS).

Citation

MLA

Strubell, E., et al. “Energy and Policy Considerations for Deep Learning in NLP”. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 3645–50, https://doi.org/10.18653/v1/P19-1355.

APA

Strubell, E., Ganesh, A., & McCallum, A. (2019). Energy and Policy Considerations for Deep Learning in NLP. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3645–3650. https://doi.org/10.18653/v1/P19-1355

Chicago

Strubell, E., A. Ganesh, and A. McCallum. 2019. “Energy and Policy Considerations for Deep Learning in NLP”. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3645–50. https://doi.org/10.18653/v1/P19-1355.

Harvard

Strubell, E., Ganesh, A. and McCallum, A. (2019) “Energy and Policy Considerations for Deep Learning in NLP”, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp. 3645–3650. Available at: https://doi.org/10.18653/v1/P19-1355.

Vancouver

1. Strubell E, Ganesh A, McCallum A (2019) Energy and Policy Considerations for Deep Learning in NLP. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp 3645–3650

BibTeX

@inproceedings{strubell-etal-2019-energy,
    title = "Energy and Policy Considerations for Deep Learning in {NLP}",
    author = "Strubell, Emma  and
      Ganesh, Ananya  and
      McCallum, Andrew",
    editor = "Korhonen, Anna  and
      Traum, David  and
      M{\`a}rquez, Llu{\'i}s",
    booktitle = "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2019",
    address = "Florence, Italy",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/P19-1355/",
    doi = "10.18653/v1/P19-1355",
    pages = "3645--3650"
}

Metadata:ACL Anthology

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Models	Hours	Cloud Cost (USD)	Electric Cost (USD)
1	120	$52–$175	$5
24	2880	$1238–$4205	$118
4789	239,942	$103k–$350k	$9870

Energy and Policy Considerations for Deep Learning in NLP

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Total Power Consumption Estimation Formula for Deep Learning Training

Knowl 2 — Estimated Energy Consumption, Carbon Footprint, and Financial Cost of Training Standard NLP Models

Knowl 3 — Carbon Dioxide Equivalent Emissions Conversion Formula for Power Consumption

Knowl 4 — Computational and Financial Cost of NLP Model Research and Development: Case Study on LISA

Knowl 5 — Comparative Carbon Footprint of NLP Model Training Against Baseline Human Activities

Knowl 6 — Hardware Power Measurement Protocol for Deep Learning Workloads

Knowl 7 — Energy Generation Profiles of Major Cloud Computing Providers and Countries

Knowl 8 — Policy and Practice Recommendations for Sustainable and Equitable NLP Research

References

Citation

BibTeX

Access the Paper