Back

Universal Sentence Encoder

Daniel CerYinfei YangSheng-yi KongNan HuaNicole LimtiacoRhomni St. JohnNoah ConstantMario Guajardo-CespedesSteve YuanChris Tar

article2018arXiv2,107 citations

Introduces the Universal Sentence Encoder, providing two pre-trained embedding architectures that balance computational efficiency with accuracy to deliver strong transfer learning performance across diverse language tasks with minimal training data.

Listen

Building accurate natural language processing applications is often hindered by the high cost and scarcity of labeled training data. While traditional transfer learning methods rely on pre-trained individual word vectors, capturing the overall meaning of full sentences usually yields better performance across diverse language tasks. This article evaluates two universal sentence encoding models to demonstrate how pre-trained sentence representations improve downstream task accuracy, reduce data labeling demands, and balance computational resource trade-offs.

The authors developed and evaluated two distinct sentence encoder architectures: a high-accuracy model based on the Transformer architecture and an efficient model based on a Deep Averaging Network (DAN). Both models convert raw English text into 512-dimensional numerical vectors and were trained using multi-task learning on web data (including Wikipedia, news articles, forums, and question-answer pages) augmented with supervised natural language inference data. The models were evaluated across diverse benchmark tasks, including customer review sentiment, subjectivity analysis, question classification, and semantic textual similarity, as well as tests assessing social bias.

The findings show that sentence-level transfer learning consistently outperforms word-level transfer baselines and models trained without pre-training. Combining sentence embeddings with word embeddings achieves the highest overall accuracy across the evaluated tasks. Notably, the sentence encoders deliver strong performance even with minimal supervision; for example, the Transformer encoder trained on only 1,000 sentiment examples rivals baseline models trained on over 67,000 examples. In terms of trade-offs, the Transformer model achieves higher overall accuracy but exhibits quadratic compute time and memory scaling as sentence length grows. Conversely, the DAN model demonstrates linear compute time and constant memory scaling, offering exceptional inference speed at only a modest reduction in accuracy. Standardized bias assessments also revealed weaker demographic and social biases in the DAN encoder compared to baseline word vectors.

These results demonstrate that adopting sentence-level embeddings substantially lowers engineering costs and accelerates deployment timelines by drastically reducing the amount of manual data annotation required. Organizations can tailor implementation to their infrastructure constraints: the Transformer model is ideal for scenarios where maximizing predictive accuracy justifies higher hardware costs and GPU usage, while the DAN model is well suited for latency-sensitive, high-throughput, or CPU-bound production environments. The authors recommend utilizing these publicly available pre-trained models to streamline language processing workflows, while actively testing for application-specific algorithmic bias before deployment in production.

Googletradetransfer learningword embeddingSentenceTransferModel
arXiv: 1803.11175
Cover for Universal Sentence Encoder

Abstract

We present models for encoding sentences into embedding vectors that specifically target transfer learning to other NLP tasks. The models are efficient and result in accurate performance on diverse transfer tasks. Two variants of the encoding models allow for trade-offs between accuracy and compute resources. For both variants, we investigate and report the relationship between model complexity, resource consumption, the availability of transfer task training data, and task performance. Comparisons are made with baselines that use word level transfer learning via pretrained word embeddings as well as baselines do not use any transfer learning. We find that transfer learning using sentence embeddings tends to outperform word level transfer. With transfer learning via sentence embeddings, we observe surprisingly good performance with minimal amounts of supervised training data for a transfer task. We obtain encouraging results on Word Embedding Association Tests (WEAT) targeted at detecting model bias. Our pre-trained sentence encoding models are made freely available for download and on TF Hub.

Topics

tradetransfer learningword embeddingcomputer resourcespart of speechsyntactic structureconvolutional neural networksword sense disambiguation

Keywords

SentenceTransferModelUseEncoderTransfer LearningModelsUniversal Sentence EncoderDANTask

Table of Contents

  • 1 Introduction
  • 2 Model Toolkit
  • 3 Encoders
  • 3.1 Transformer
  • 3.2 Deep Averaging Network (DAN)
  • 3.3 Encoder Training Data
  • 4 Transfer Tasks
  • 5 Transfer Learning Models
  • 5.1 Baselines
  • 5.2 Combined Transfer Models
  • 6 Experiments
  • 7 Results
  • 7.1 Discussion
  • 8 Resource Usage
  • 9 Conclusion
  • References

Knowls

  1. Knowl 1 — Transformer-Based Universal Sentence Encoder

    model/method

    The Transformer-based Universal Sentence Encoder (USE_T) generates fixed-length 512-dimensional vector embeddings from text strings tokenized using lowercased Penn Treebank (PTB) tokenization. The model employs the encoding sub-graph of the Transformer architecture, using self-attention mechanisms to construct context-aware representations for all tokens in a sentence that account for token ordering and global sentence context.

    To produce a fixed-dimensional sentence embedding vector e∈R512\mathbf{e} \in \mathbb{R}^{512} from the variable-length sequence of token representations h1,h2,…,hL∈R512\mathbf{h}_1, \mathbf{h}_2, \dots, \mathbf{h}_L \in \mathbb{R}^{512}, the encoder computes the element-wise sum of the representations across all token positions and divides by the square root of the sentence length LL:

    e=∑i=1LhiL\mathbf{e} = \frac{\sum_{i=1}^{L} \mathbf{h}_i}{\sqrt{L}}

    Dividing by L\sqrt{L} prevents the difference between short and long sentence embeddings from being dominated by sentence length effects.

    The model is trained within a multi-task learning framework to support general-purpose downstream transfer. Its computational time and memory consumption both scale quadratically, O(L2)O(L^2), with sequence length LL.

  2. Knowl 2 — Deep Averaging Network Universal Sentence Encoder

    model/method

    The Deep Averaging Network (DAN) variant of the Universal Sentence Encoder (USE_D) maps lowercased PTB-tokenized text strings into 512-dimensional sentence embeddings while ensuring linear compute time and constant memory scaling during inference.

    The encoding process consists of three steps:

    1. Input embeddings are looked up for all constituent words (unigrams) as well as word bi-grams in the sentence.
    2. The unigram and bi-gram embedding vectors are averaged element-wise into a single composite representation vector.
    3. The averaged vector is passed through a feedforward deep neural network (DNN) with multiple dense layers to produce the final 512-dimensional sentence embedding.

    The DAN encoder is trained using the same multi-task training setup as the Transformer-based encoder. The primary benefit of the DAN architecture is computational efficiency: its compute time is O(L)O(L) in sentence length LL, and its inference memory footprint remains constant with sequence length.

  3. Knowl 3 — Multi-Task Training Formulation and Data Sources for Sentence Encoders

    model/method

    The Universal Sentence Encoders (both Transformer and DAN variants) are trained using a multi-task objective that combines unsupervised and supervised learning tasks across diverse corpora:

    1. Modified Skip-Thought Task: An unsupervised objective in which the encoder consumes running text and predicts context sentences (the preceding and subsequent sentences). In the Transformer model, the LSTM of the standard Skip-Thought architecture is replaced by the Transformer encoder sub-graph.
    2. Conversational Input-Response Task: An objective that scores and selects correct response candidates for conversational context inputs, trained on parsed dialogue data.
    3. Supervised Natural Language Inference (SNLI): A supervised three-way classification task on the Stanford Natural Language Inference dataset (entailment, contradiction, neutral), which substantially improves downstream transfer performance.

    Unsupervised training data is sourced from Wikipedia, web news, question-answer websites, and web discussion forums, augmented by the supervised SNLI corpus.

  4. Knowl 4 — Angular Similarity Metric for Sentence Embeddings

    equation

    For assessing the pairwise semantic textual similarity between two sentence embedding vectors u,v∈Rd\mathbf{u}, \mathbf{v} \in \mathbb{R}^d, raw cosine similarity is converted into an angular distance metric normalized to the interval [0,1][0, 1]:

    sim(u,v)=1−arccos⁡(u⋅v∥u∥∥v∥)π\text{sim}(\mathbf{u}, \mathbf{v}) = 1 - \frac{\arccos\left(\frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\| \|\mathbf{v}\|}\right)}{\pi}

    where u⋅v\mathbf{u} \cdot \mathbf{v} is the standard dot product, ∥u∥\|\mathbf{u}\| and ∥v∥\|\mathbf{v}\| are the Euclidean norms (L2L_2 norms) of the vectors, and arccos⁡(⋅)\arccos(\cdot) outputs an angle in radians in the range [0,π][0, \pi].

    Using angular similarity derived from the arc cosine of normalized cosine similarity consistently yields higher Pearson correlation (rr) with human similarity judgments on the Semantic Textual Similarity (STS) Benchmark than using raw cosine similarity directly.

  5. Knowl 5 — Transfer Task Performance Across NLP Benchmarks

    data/table

    The transfer performance of the Transformer encoder (USE_T) and DAN encoder (USE_D) was evaluated on MR (movie reviews), CR (customer reviews), SUBJ (subjectivity), MPQA (opinion polarity), TREC (question classification), SST (sentiment classification), and the STS Benchmark (semantic similarity). Models are compared against word-level transfer using pre-trained word2vec skip-gram embeddings (w2v w.e.), randomly initialized word embeddings learned only on task data (lrn w.e.), and baseline CNN / DAN architectures.

    Model MR CR SUBJ MPQA TREC SST STS Bench (dev / test)
    Sentence Word Embedding Transfer Learning
    USE_D+DAN (w2v w.e.) 77.11 81.71 93.12 87.01 94.72 82.14 –
    USE_D+CNN (w2v w.e.) 78.20 82.04 93.24 85.87 97.67 85.29 –
    USE_T+DAN (w2v w.e.) 81.32 86.66 93.90 88.14 95.51 86.62 –
    USE_T+CNN (w2v w.e.) 81.18 87.45 93.58 87.32 98.07 86.69 –
    Sentence Embedding Transfer Learning
    USE_D 74.45 80.97 92.65 85.38 91.19 77.62 0.763 / 0.719 (r)
    USE_T 81.44 87.43 93.87 86.98 92.51 85.38 0.814 / 0.782 (r)
    USE_D+DAN (lrn w.e.) 77.57 81.93 92.91 85.97 95.86 83.41 –
    USE_D+CNN (lrn w.e.) 78.49 81.49 92.99 85.53 97.71 85.27 –
    USE_T+DAN (lrn w.e.) 81.36 86.08 93.66 87.14 96.60 86.24 –
    USE_T+CNN (lrn w.e.) 81.59 86.45 93.36 86.85 97.44 87.21 –
    Word Embedding Transfer Learning
    DAN (w2v w.e.) 74.75 75.24 90.80 81.25 85.69 80.24 –
    CNN (w2v w.e.) 75.10 80.18 90.84 81.38 97.32 83.74 –
    Baselines with No Transfer Learning
    DAN (lrn w.e.) 75.97 76.91 89.49 80.93 93.88 81.52 –
    CNN (lrn w.e.) 76.39 79.39 91.18 82.20 95.82 84.90 –

    Classification results report accuracy (%) averaged over 10 runs; STS Bench reports Pearson correlation (rr) computed via angular similarity on sentence embeddings. Transfer learning with sentence embeddings consistently surpasses word-level transfer alone. USE_T generally outperforms USE_D, while combining sentence embeddings with word embeddings achieves the highest overall accuracy.

  6. Knowl 6 — Sentence Embedding Sample Efficiency Under Limited Training Data

    data/table

    The effect of training set size on transfer performance was evaluated on the Stanford Sentiment Treebank (SST) by sub-sampling training data from 1,000 examples (1k) to the full training set of 67,349 examples (67.3k).

    Model SST 1k SST 2k SST 4k SST 8k SST 16k SST 32k SST 67.3k
    Sentence Word Embedding Transfer Learning
    USE_D+DNN (w2v w.e.) 78.65 78.68 79.07 81.69 81.14 81.47 82.14
    USE_D+CNN (w2v w.e.) 77.79 79.19 79.75 82.32 82.70 83.56 85.29
    USE_T+DNN (w2v w.e.) 85.24 84.75 85.05 86.48 86.44 86.38 86.62
    USE_T+CNN (w2v w.e.) 84.44 84.16 84.77 85.70 85.22 86.38 86.69
    Sentence Embedding Transfer Learning
    USE_D 77.47 76.38 77.39 79.02 78.38 77.79 77.62
    USE_T 84.85 84.25 85.18 85.63 85.83 85.59 85.38
    USE_D+DNN (lrn w.e.) 75.90 78.68 79.01 82.31 82.31 82.14 83.41
    USE_D+CNN (lrn w.e.) 77.28 77.74 79.84 81.83 82.64 84.24 85.27
    USE_T+DNN (lrn w.e.) 84.51 84.87 84.55 85.96 85.62 85.86 86.24
    USE_T+CNN (lrn w.e.) 82.66 83.73 84.23 85.74 86.06 86.97 87.21
    Word Embedding Transfer Learning
    DNN (w2v w.e.) 66.34 69.67 73.03 77.42 78.29 79.81 80.24
    CNN (w2v w.e.) 68.10 71.80 74.91 78.86 80.83 81.98 83.74
    Baselines with No Transfer Learning
    DNN (lrn w.e.) 66.87 71.23 73.70 77.85 78.07 80.15 81.52
    CNN (lrn w.e.) 67.98 71.81 74.90 79.14 81.04 82.72 84.90

    With only 1,000 training examples, transfer learning with USE_T reaches 84.85% accuracy (or 85.24% when paired with a task DNN), matching or outperforming non-transfer CNN (84.90%) and DNN (81.52%) models trained on the complete 67.3k dataset. As the training dataset grows, models trained without transfer learning approach the accuracy of transfer models, showing that sentence embedding transfer provides the largest gains when supervised data is scarce.

  7. Knowl 7 — Compute and Memory Resource Scaling of Transformer vs. DAN Encoders

    empirical result

    The Transformer (USE_T) and Deep Averaging Network (USE_D) encoders present distinct resource usage trade-offs across sentence lengths:

    1. Compute Time: USE_T has computational complexity O(n2)O(n^2) with sentence length nn, while USE_D has O(n)O(n) complexity. On short sentences, USE_T is only moderately slower than USE_D, but its compute time increases dramatically as sentence length grows. In contrast, USE_D compute time remains nearly constant with length. GPU acceleration provides a substantially larger practical runtime reduction for USE_T than for USE_D.
    2. Memory Scaling: USE_T space complexity scales quadratically O(n2)O(n^2) with sentence length, whereas USE_D memory usage is constant O(1)O(1) with sentence length.
    3. Baseline Parameter Footprint: USE_D memory is dominated by storing static unigram and bigram embedding tables. Because USE_T stores only unigram embeddings and computes compositions dynamically, USE_T consumes approximately half as much memory as USE_D on short sequences, before its quadratic scaling increases memory on longer sequences.

  8. Knowl 8 — Model Bias Characterization via Word Embedding Association Tests

    data/table

    Model bias in the Deep Averaging Network sentence encoder (USE_D) was evaluated using the Word Embedding Association Test (WEAT) and compared against pre-trained GloVe word embeddings. Effect sizes are measured as Cohen's dd over mean cosine similarities across grouped attribute words, with 1-tailed pp-values.

    Target words Attrib. words GloVe Uni. Enc. (DAN)
    dd pp dd pp
    Eur.-American vs Afr.-American names Pleasant vs. Unpleasant 1 1.41 10−810^{-8} 0.361 0.035
    Eur.-American vs. Afr.-American names Pleasant vs. Unpleasant from (a) 1.50 10−410^{-4} -0.372 0.87
    Eur.-American vs. Afr.-American names Pleasant vs. Unpleasant from (c) 1.28 10−310^{-3} 0.721 0.015
    Male vs. female names Career vs family 1.81 10−310^{-3} 0.0248 0.48
    Math vs. arts Male vs. female terms 1.06 0.018 0.588 0.12
    Science vs. arts Male vs female terms 1.24 10−210^{-2} 0.236 0.32
    Mental vs. physical disease Temporary vs permanent 1.38 10−210^{-2} 1.60 0.0027
    Young vs old peoples names Pleasant vs unpleasant 1.21 10−210^{-2} 1.01 0.022
    Flowers vs. insects Pleasant vs. Unpleasant 1.50 10−710^{-7} 1.38 10−710^{-7}
    Instruments vs. Weapons Pleasant vs Unpleasant 1.53 10−710^{-7} 1.44 10−710^{-7}

    The universal encoder replicates standard human cognitive associations (e.g., Flowers vs. Insects and Instruments vs. Weapons with p=10−7p = 10^{-7}). On probes measuring demographic biases—including racial name associations, gender stereotypes (career vs. family, math/science vs. arts), and ageism—the encoder exhibits substantially smaller effect sizes (dd) and reduced statistical significance compared to GloVe embeddings.

Coverage note — No substantial contributed material was omitted. TF Hub deployment URLs and Python usage code snippets were omitted as standard software distribution mechanics.

References

  1. 1.Martın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. Tensorflow: A system for large-scale machine learning. In Proceedings of USENIX OSDI'16.
  2. 2.Marianne Bertrand and Sendhil Mullainathan. 2004. Are emily and greg more employable than lakisha and jamal? a field experiment on labor market discrimination. The American Economic Review, 94(4).
  3. 3.Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of EMNLP.
  4. 4.Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186.
  5. 5.Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of SemEval-2017.
  6. 6.Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364.
  7. 7.Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D. Sculley. Google vizier: A service for black-box optimization. In Proceedings of KDD '17.
  8. 8.Anthony G. Greenwald, Debbie E. McGhee, and Jordan L. K. Schwartz. 1998. Measuring individual differences in implicit cognition: the implicit association test. Journal of personality and social psychology, 74(6).
  9. 9.Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, Laszlo Lukacs, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. CoRR, abs/1705.00652.
  10. 10.Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735–1780.
  11. 11.Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of KDD '04.
  12. 12.Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daume III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proceedings of ACL/IJCNLP.
  13. 13.Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP.
  14. 14.Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In In Proceedings of NIPS.
  15. 15.Xin Li and Dan Roth. 2002. Learning question classifiers. In Proceedings of COLING '02.
  16. 16.Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS'13.
  17. 17.Lindsey L. Monteith and Jeremy W. Pettit. 2011. Implicit and explicit stigmatizing attitudes and stereotypes about depression. Journal of Social and Clinical Psychology, 30(5).
  18. 18.Brian A. Nosek, Mahzarin R. Banaji, and Anthony G. Greenwald. 2002a. Harvesting implicit group attitudes and beliefs from a demonstration web site. Group Dynamics, 6(1).
  19. 19.Brian A. Nosek, Mahzarin R. Banaji, and Anthony G Greenwald. 2002b. Math = male, me = female, therefore math me. Journal of Personality and Social Psychology,, 83(1).
  20. 20.Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL'04), Main Volume.
  21. 21.Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of ACL'05.
  22. 22.Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceeding of EMNLP.
  23. 23.Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP.
  24. 24.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS.
  25. 25.Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2):165–210.

Citation

MLA
Cer, D., et al. “Universal Sentence Encoder”. arXiv, 2018, http://arxiv.org/abs/1803.11175v2.
APA
Cer, D., Yang, Y., Kong, S.-. yi ., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., Sung, Y.-H., Strope, B., & Kurzweil, R. (2018). Universal Sentence Encoder. arXiv. http://arxiv.org/abs/1803.11175v2
Chicago
Cer, D., Y. Yang, S.-. yi . Kong, et al. 2018. “Universal Sentence Encoder”. arXiv. http://arxiv.org/abs/1803.11175v2.
Harvard
Cer, D. et al. (2018) “Universal Sentence Encoder”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1803.11175v2.
Vancouver
1. Cer D, Yang Y, Kong S-yi, et al (2018) Universal Sentence Encoder. arXiv

BibTeX

@article{cer2018universal,
  title = {Universal Sentence Encoder},
  author = {Cer, Daniel and Yang, Yinfei and Kong, Sheng-yi and Hua, Nan and Limtiaco, Nicole and John, Rhomni St. and Constant, Noah and Guajardo-Cespedes, Mario and Yuan, Steve and Tar, Chris and Sung, Yun-Hsuan and Strope, Brian and Kurzweil, Ray},
  year = {2018},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1803.11175v2},
  eprint = {1803.11175}
}
Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Published with permission