BLEURT: Learning Robust Metrics for Text Generation

Thibault SellamDipanjan DasAnkur P. Parikh

article2020ACL1,998 citations

Proposes BLEURT, a learned text generation evaluation metric that uses synthetic pre-training on BERT to closely model human judgments across benchmarks even when evaluation data is scarce or out-of-distribution.

Listen

Natural language generation systems have advanced significantly, but standard automatic evaluation metrics have failed to keep pace. Traditional surface-level metrics such as BLEU and ROUGE rely heavily on exact word overlaps and frequently correlate poorly with human quality judgments. While human evaluations provide the gold standard, collecting manual ratings is too expensive and slow for continuous model development pipelines. Recent machine learning-based metrics offer improved alignment with human scores, but they often struggle to generalize across new domains or fail when systems improve over time—a scenario known as distribution and quality drift.

The article evaluates whether a fully learned evaluation metric, named BLEURT (Bilingual Evaluation Understudy with Representations from Transformers), can achieve high correlation with human judgment and remain robust across varying data distributions by incorporating synthetic pre-training before fine-tuning on human ratings.

To address this, the researchers developed a novel multi-stage training pipeline. Starting with a pre-trained contextual language representation model (BERT), the authors introduced an intermediate pre-training phase using approximately 6.5 million synthetic sentence pairs generated from 1.8 million Wikipedia segments. These pairs were created through BERT-based token masking, backtranslation across languages, and random word dropping to mimic common generation errors and semantic variations. The model was trained on nine diverse lexical and semantic supervision tasks—including existing automated metrics, translation likelihoods, and textual entailment—before fine-tuning on relatively small collections of human-rated quality datasets. The approach was tested across multiple benchmarks, including several years of the Workshop on Machine Translation (WMT) Metrics Shared Task (2017–2019) and the WebNLG data-to-text challenge.

The analysis produced several critical findings. First, BLEURT achieved state-of-the-art agreement with human ratings across three years of the WMT benchmark, consistently outperforming standard metrics and prior competitive learned baselines across multiple language pairs. Second, the synthetic pre-training step provided substantial performance gains, adding up to 7.4 Kendall's Tau correlation points on translation benchmarks compared to training without it, with particularly large benefits observed for smaller model architectures and constrained training sets. Third, in synthetic stress tests measuring quality drift—where models trained only on low-quality outputs had to evaluate high-quality outputs—BLEURT maintained strong correlation, whereas models without synthetic pre-training degraded rapidly and fell below basic baseline metrics. Fourth, on the WebNLG benchmark, pre-trained BLEURT adapted efficiently to new generation tasks with as few as 800 training examples, whereas models lacking synthetic pre-training required significantly more data to achieve comparable performance. Finally, ablation studies showed that pre-training on higher-level semantic signals (such as BERTscore, translation likelihood, and entailment) drove performance gains, whereas pre-training on simple surface metrics like BLEU and ROUGE degraded accuracy.

These findings indicate that synthetic pre-training effectively solves the primary trade-off between model expressiveness and robustness in text generation evaluation. Organizations developing automated text systems can reduce reliance on slow, costly human evaluations without sacrificing evaluation reliability or risking catastrophic failure when underlying generation quality shifts. Relying on outdated surface-overlap metrics creates a false sense of security and risks misguiding model optimization.

Organizations evaluating language generation systems should transition toward robust, pre-trained evaluation models like BLEURT for benchmarking and development. For domain-specific applications, practitioners should leverage BLEURT’s pre-trained checkpoints, which require minimal task-specific human annotations to reach high reliability. Future development should focus on refining the composition of synthetic pre-training tasks—prioritizing semantic and entailment objectives over simple overlap heuristics—and extending the approach to non-English languages.

Confidence in these findings is high across English translation and data-to-text domains due to extensive validation across multiple standard benchmark years and controlled stress tests. However, caution is warranted when evaluating languages other than English, as the current implementation and evaluation were restricted to English-target outputs. Additionally, while BLEURT exhibits strong resilience under moderate quality drift, extreme distribution shifts where train and test domains share almost no overlap still show reduced correlation performance.

Google complexity metrics human evaluation BLEURT Metrics Data

arXiv: 2004.04696 google-research/bleurt

Paper: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Jacob Devlin et al. (2019). BLEURT directly builds upon the BERT architecture and contextual representations as the core backbone for its learned evaluation model.
Paper: BERTScore: Evaluating Text Generation with BERT, Tianyi Zhang et al. (2019). BERTScore introduced evaluating text generation using contextual BERT embeddings, providing the foundational conceptual paradigm that BLEURT refines via synthetic pre-training.
Paper: Bleu: a Method for Automatic Evaluation of Machine Translation, Kishore Papineni et al. (2002). This paper establishes the foundational BLEU metric for automatic evaluation, the primary standard whose limitations with human correlation BLEURT is designed to address.
Paper: ROUGE: A Package for Automatic Evaluation of Summaries, Chin-Yew Lin (2004). This work defines ROUGE, the ubiquitous reference-based n-gram evaluation metric that serves alongside BLEU as a key baseline motivating BLEURT.
Paper: RoBERTa: A Robustly Optimized BERT Pretraining Approach, Yinhan Liu et al. (2019). RoBERTa refines BERT pre-training strategies that inform modern contextual transformer representations utilized in learned metric architectures.

Paper: G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment, Yang Liu et al. (2023). G-Eval advances beyond encoder-based learned metrics like BLEURT by leveraging modern large language models and chain-of-thought prompting for human-aligned text generation evaluation.
Paper: From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge, Dawei Li et al. (2025). This survey examines the broader paradigm shift from dedicated learned metrics like BLEURT to generalized LLM-as-a-judge evaluation frameworks.
Paper: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Lianmin Zheng et al. (2023). This work operationalizes LLM-based evaluation across multi-turn generation benchmarks, extending the pursuit of automated evaluation beyond traditional reference matching.
Paper: Ragas: Automated Evaluation of Retrieval Augmented Generation, Shahul Es et al. (2024). Ragas builds on the evolution of automated generation evaluation by assessing multidimensional attributes in retrieval-augmented generation pipelines.
Paper: A Survey on Evaluation of Large Language Models, Yu-Chu Chang et al. (2023). This survey provides a comprehensive overview of the modern language model evaluation landscape, contextualizing learned metrics within wider automated testing methodologies.

Cover for BLEURT: Learning Robust Metrics for Text Generation

Abstract

Text generation has made significant advances in the last few years. Yet, evaluation metrics have lagged behind, as the most popular choices (e.g., BLEU and ROUGE) may correlate poorly with human judgments. We propose BLEURT, a learned evaluation metric based on BERT that can model human judgments with a few thousand possibly biased training examples. A key aspect of our approach is a novel pre-training scheme that uses millions of synthetic examples to help the model generalize. BLEURT provides state-of-the-art results on the last three years of the WMT Metrics shared task and the WebNLG Competition dataset. In contrast to a vanilla BERT-based approach, it yields superior results even when the training data is scarce and out-of-distribution.

Topics

complexity metrics human evaluation

Keywords

BLEURT Metrics Data Pre-training WMT Training Task Proceedings Generation Use

1 Introduction
2 Preliminaries
3 Fine-Tuning BERT for Quality Evaluation
4 Pre-Training on Synthetic Data
4.1 Generating Sentence Pairs
4.2 Pre-Training Signals
4.3 Modeling
5 Experiments
5.1 WMT Metrics Shared Task
5.2 Robustness to Quality Drift
5.3 WebNLG Experiments
5.4 Ablation Experiments
6 Related Work
7 Conclusion
References
A Implementation Details of the Pre-Training Phase
A.1 Data Generation
A.2 Pre-Training Tasks
A.3 Modeling
B Experiments–Supplementary Material
B.1 Training Setup for All Experiments
B.2 WMT Metric Shared Task
B.3 Robustness to Quality Drift
B.4 Ablation Experiment–How Much Pre-Training Time is Necessary?

Knowls

Knowl 1 — BLEURT Model Architecture and Three-Stage Training Pipeline
model/method
BLEURT (Bilingual Evaluation Understudy with Representations from Transformers) is a learned, reference-based sentence-level metric for natural language generation (NLG) evaluation. Given a reference sentence $x = (x_1, \dots, x_r)$ and a model-generated prediction sentence $\tilde{x} = (\tilde{x}_1, \dots, \tilde{x}_p)$ , BLEURT predicts a scalar score $\hat{y} \in \mathbb{R}$ reflecting human quality assessment.

BLEURT models are trained via a three-stage pipeline:
1. Standard Transformer Pre-training: Initialization from standard unsupervised pre-trained BERT checkpoints (BERT-Base with 12 layers, 768 hidden units, 12 attention heads, or BERT-Large with 24 layers, 1024 hidden units, 16 attention heads).
2. Multi-Task Synthetic Pre-training: Intermediate pre-training on millions of synthetic sentence pairs $(z, \tilde{z})$ generated from perturbed Wikipedia sentences, optimized with a multi-task objective covering lexical, semantic, and probabilistic supervision signals.
3. Task-Specific Fine-Tuning: Supervised fine-tuning on human rating datasets (e.g., translation assessment or data-to-text ratings) using a mean squared error regression loss.
At inference, the reference $x$ and candidate $\tilde{x}$ are concatenated as input to BERT, and a linear layer on the special $[\text{CLS}]$ contextual vector $v_{[\text{CLS}]}$ outputs the predicted quality rating.
Knowl 2 — Multi-Task Synthetic Pre-Training Objectives and Loss Formulation
equation
During the synthetic pre-training phase, BLEURT is optimized over $M$ synthetic sentence pairs $(z^m, \tilde{z}^m)$ across $K$ pre-training tasks using a weighted sum of task-specific losses:

$\ell_{\text{pre-training}} = \frac{1}{M} \sum_{m=1}^M \sum_{k=1}^K \gamma_k \ell_k(\tau_k^m, \hat{\tau}_k^m)$

where $\tau_k^m$ is the ground-truth target vector for task $k$ on synthetic example $m$ , $\hat{\tau}_k^m$ is the model's prediction, and $\gamma_k \ge 0$ is a task weight hyperparameter.

For a regression task $k$ with target dimension $|\tau_k|$ , the loss is the normalized squared $\ell_2$ error:

$\ell_k(\tau_k, \hat{\tau}_k) = \frac{\|\tau_k - \hat{\tau}_k\|_2^2}{|\tau_k|}, \quad \text{with } \hat{\tau}_k = W_{\tau_k} v_{[\text{CLS}]} + b_{\tau_k}$

where $W_{\tau_k}$ and $b_{\tau_k}$ are the task-specific linear projection parameters applied to the $[\text{CLS}]$ contextualized embedding $v_{[\text{CLS}]}$ . All continuous regression targets are normalized prior to training.

For a multiclass classification task $k$ , the model computes class logits $\hat{\tau}_{kc} = W_{\tau_{kc}} v_{[\text{CLS}]} + b_{\tau_{kc}}$ for each class $c$ and optimizes the standard multiclass cross-entropy loss.
Knowl 3 — Pre-Training Supervision Signals for Synthetic Metric Tuning
definition
BLEURT employs 9 distinct supervision signals across 6 task types during its synthetic pre-training phase on sentence pairs $(z, \tilde{z})$ :
1. Sentence BLEU ( $\tau_{\text{BLEU}}$ ): Standard smoothed sentence-level BLEU score (regression).
2. ROUGE ( $\tau_{\text{ROUGE}}$ ): A 3-dimensional target vector $(\tau_{\text{ROUGE-P}}, \tau_{\text{ROUGE-R}}, \tau_{\text{ROUGE-F}})$ containing ROUGE-N precision, recall, and F-measure (regression).
3. BERTscore ( $\tau_{\text{BERTscore}}$ ): A 3-dimensional target vector $(\tau_{\text{BERTscore-P}}, \tau_{\text{BERTscore-R}}, \tau_{\text{BERTscore-F}})$ containing BERTscore precision, recall, and F-score computed with BERT-Large (regression).
4. Backtranslation Log-Likelihoods: 4 directional regression signals measuring translation probability normalized by sequence length across English-French and English-German pairs:
$\tau_{en\text{-}fr, \tilde{z}|z} = \frac{\log P(\tilde{z}|z)}{|\tilde{z}|} \approx \frac{\log P_{fr\to en}(\tilde{z}|z^*_{fr})}{|\tilde{z}|}, \quad \text{where } z^*_{fr} = \arg\max_{z_{fr}} P_{en\to fr}(z_{fr}|z)$

and analogously for $\tau_{en\text{-}fr, z|\tilde{z}}$ , $\tau_{en\text{-}de, \tilde{z}|z}$ , and $\tau_{en\text{-}de, z|\tilde{z}}$ . 5. Textual Entailment ( $\tau_{\text{entail}}$ ): Probabilities over 3 classes $(\tau_{\text{Entail}}, \tau_{\text{Contradict}}, \tau_{\text{Neutral}})$ generated by a BERT model fine-tuned on the Multi-Genre Natural Language Inference (MNLI) corpus (multiclass classification). 6. Backtranslation Flag ( $\tau_{\text{backtran\_flag}}$ ): A binary label indicating whether the synthetic candidate $\tilde{z}$ was generated via backtranslation or BERT mask-filling (multiclass/binary classification).
Knowl 4 — Generation of Synthetic Reference-Candidate Sentence Pairs
model/method
To prepare the evaluation metric for diverse lexical, syntactic, and semantic distortions without access to target generation systems, synthetic sentence pairs $(z, \tilde{z})$ are automatically generated by perturbing $1.8$ million English Wikipedia segments $z$ , yielding approximately $6.5$ million perturbed candidates $\tilde{z}$ .

Three perturbation strategies are utilized:
1. Mask-Filling with BERT: Masks are inserted into Wikipedia sentences and filled using BERT's masked language modeling head via beam search (beam size 8) to maintain fluency while introducing lexical alterations. Two masking patterns are applied with up to 15 masks per sentence: randomly scattered masked positions, or a single contiguous segment spanning words between sampled start position $s$ and length $l$ .
2. Backtranslation: Round-trip translations (English $\to$ French/German $\to$ English) using sequence-to-sequence Transformer models generate semantic paraphrases as well as realistic neural MT errors.
3. Random Word Dropout: Random words are uniformly dropped from $\tilde{z}$ (applied to approximately $30\%$ of the generated pairs) to expose the metric to pathological NLG phenomena such as word omissions, truncations, and empty outputs.
Knowl 5 — Supervised Fine-Tuning on Human Quality Judgments
equation
Following synthetic pre-training, BLEURT is fine-tuned on a dataset of $N$ human-rated examples $\{(x_i, \tilde{x}_i, y_i)\}_{i=1}^N$ , where $x_i$ is the reference sentence, $\tilde{x}_i$ is the candidate prediction, and $y_i \in \mathbb{R}$ is the human rating.

The predicted score $\hat{y} = f(x, \tilde{x})$ is parameterized by a linear regression layer applied to the final Transformer $[\text{CLS}]$ representation $v_{[\text{CLS}]}$ :

$\hat{y} = W v_{[\text{CLS}]} + b$

where $W$ is the linear weight matrix and $b$ is the bias vector.

Both the linear projection layer and the underlying BERT parameters are optimized end-to-end using the mean squared error regression loss:

$\ell_{\text{supervised}} = \frac{1}{N} \sum_{n=1}^N \|y_n - \hat{y}_n\|^2$

Fine-tuning is conducted using the Adam optimizer with a learning rate of $1\times 10^{-5}$ , batch size 32, and trained for 40,000 steps with validation checkpointing every 1,500 steps.

Knowl 6 — Evaluation Performance on WMT Metrics Shared Tasks (2017–2019)

data/table

BLEURT was evaluated on the to-English language pairs of the WMT17, WMT18, and WMT19 Metrics Shared Tasks. Agreement with human quality ratings is measured using Kendall's Tau ( $\tau$ ) and Pearson correlation ( $r$ ) for WMT17, and Kendall's Tau ( $\tau$ ) and Direct Assessment ( $DA$ , a robust Kendall's Tau variant with a 25-point similarity filter) for WMT18 and WMT19. All values are scaled by $\times 100$ .

Metric	WMT17 Avg		WMT18 Avg		WMT19 Avg
	$\tau$	$r$	$\tau$	$DA$	$\tau$	$DA$
sentBLEU	32.4	47.5	22.8	23.2	22.7	22.3
BERTscore (BERT-Large)	53.3	73.2	31.0	35.7	30.0	32.1
BERTscore (RoBERTa)	58.0	76.8	32.9	37.1	32.0	33.6
chrF++	39.6	57.9	–	–	–	–
BEER	39.1	57.1	–	–	–	–
RUSE	–	–	28.6	32.4	–	–
YiSi-1	–	–	26.3	30.4	–	–
ESIM	–	–	–	–	31.6	31.3
BLEURTbase -pre (No synth pretrain)	56.8	75.8	33.6	36.4	32.3	31.7
BLEURTbase	61.0	80.2	34.9	37.4	33.6	32.9
BLEURT -pre (No synth pretrain)	59.8	79.2	34.5	37.0	33.2	32.9
BLEURT (BERT-Large)	62.5	81.8	35.6	38.1	33.8	33.6

Across all benchmark years, BLEURT achieves state-of-the-art correlation with human assessments. Pre-training on synthetic data consistently improves performance over direct fine-tuning (-pre), with the largest gain on WMT17 where human training data is smallest ( $5,360$ records).

Knowl 7 — Robustness of BLEURT to Quality and Distribution Drift
empirical result
To evaluate resilience to quality drift (where generation systems improve over time and ratings shift towards higher quality), a controlled sub-sampling protocol was constructed on WMT17 by partitioning ratings into 10 quantile bins $B \in \{1, \dots, 10\}$ and sampling training instances with probability $B^{-\alpha}$ (favoring low-quality predictions) and test instances with probability $(11 - B)^{-\alpha}$ (favoring high-quality predictions), parameterized by a skew factor $\alpha \in \{0, 0.5, 1.0, 1.5, 3.0\}$ .

Key empirical findings:
- Vanilla BERT Degradation: Direct fine-tuning of BERT on human ratings without synthetic pre-training (BLEURT -pre) is highly vulnerable to distribution skew: its correlation drops below BERTscore at $\alpha = 1.0$ and falls below surface sentBLEU at $\alpha \ge 1.5$ .
- Synthetic Pre-Training Resilience: BLEURT with synthetic pre-training maintains high Kendall Tau correlations across varying skew levels. It only drops below baseline metrics under extreme drift ( $\alpha = 3.0$ ), where training data shrinks to only $11.9\%$ of original data ( $636$ samples) and the training and test rating distributions are almost completely non-overlapping.
Knowl 8 — Sample Efficiency and Few-Shot Adaptation on WebNLG
empirical result
BLEURT's cross-domain transfer capability was evaluated on the WebNLG 2017 dataset across three evaluation criteria: semantics, grammar, and fluency, using splits varying from 0 training instances up to $\sim 50\%$ of the data ( $2,689$ records), split either by RDF input triples or by evaluated generation systems.

Key empirical results:
- Zero-Shot Transfer: BLEURT pre-trained on synthetic data and intermediate-tuned on WMT translation ratings achieves correlation competitive with BERTscore on WebNLG without any WebNLG training records.
- Few-Shot Adaptation: BLEURT pre-trained only on synthetic data (BLEURT -wmt) becomes competitive with established baselines (BLEU, TER, Meteor, BERTscore) with as few as 836 training records.
- Failure of Direct Fine-Tuning: Vanilla BERT trained directly from public checkpoints (BLEURT -pre -wmt) requires at least one-third of the dataset to outperform standard baselines on most tasks, and continues to lag behind on semantic evaluations under system splits.
Knowl 9 — Impact and Ablation of Individual Pre-Training Signals
empirical result
Ablation experiments on the WMT17 benchmark evaluating both single-signal pre-training and leave-one-task-out pre-training demonstrate the relative utility of pre-training objectives:
- Positive Contributors: Pre-training on neural and semantic signals—specifically BERTscore, MNLI textual entailment probabilities, and backtranslation likelihoods—provides the largest improvements in downstream Kendall Tau correlation. Symmetrically, removing any of these signals during pre-training causes measurable degradation in evaluation performance.
- Negative Impact of N-gram Overlap: Pre-training on surface lexical metrics (BLEU and ROUGE) slightly degrades BLEURT's final correlation with human judgments relative to pre-training without them. This indicates that pre-training on objectives that correlate poorly with human judgment can bias the learned representations away from semantic fidelity.
Knowl 10 — Pre-Training Efficiency and Convergence Schedule
empirical result
Evaluating downstream Kendall's Tau on the WMT17 benchmark as a function of pre-training step count (from 0 to 800,000 steps at batch size 32) indicates that synthetic pre-training is highly sample-efficient:
- The majority of performance improvements are achieved within the first 400,000 steps (equivalent to approximately 2 epochs over the 6.5 million synthetic examples).
- At 400,000 steps, BLEURTbase improves its Kendall Tau agreement from 0.555 (0 steps) to over 0.610 on WMT17, with diminishing marginal gains observed between 400,000 and 800,000 steps.

Coverage note — None. All primary contributions—including the BLEURT architecture, synthetic generation strategies, multi-task pre-training formulation, WMT benchmarks, quality drift stress-tests, WebNLG domain transfer, and ablations—are fully covered.

References

1.Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR.
2.Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of ACL.
3.Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In Proceedings of ICLR.
4.Ondřej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the wmt17 metrics shared task. In Proceedings of WMT.
5.Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016. Results of the wmt16 metrics shared task. In Proceedings of WMT.
6.Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. Proceedings of EMNLP.
7.Arun Tejasvi Chaganty, Stephen Mussman, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evaluation. Proceedings of ACL.
8.Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced lstm for natural language inference. Proceedings of ACL.
9.Sumit Chopra, Michael Auli, and Alexander M Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of NAACL HLT.
10.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL HLT.
11.Ondřej Dušek, Karin Sevegnani, Ioannis Konstas, and Verena Rieser. 2019. Automatic quality estimation for natural language generation: Ranting (jointly rating and ranking). Proceedings of INLG.
12.Matan Eyal, Tal Baumel, and Michael Elhadad. 2019. Question answering as an automatic evaluation metric for news article summarization. In Proceedings of NAACL HLT.
13.Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. 2015. From captions to visual concepts and back. In Proceedings of CVPR.
14.Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. Ppdb: The paraphrase database. In Proceedings NAACL HLT.
15.Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The webnlg challenge: Generating text from rdf data. In Proceedings of INLG.
16.Ben Goodrich, Mohammad Ahmad Saleh, Peter Liu, and Vinay Rao. 2019. Assessing the factual accuracy of text generation. In Proceedings of ACM SIGKDD.
17.Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. Proceedings of NAACL HLT.
18.Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. Proceedings of EMNLP.
19.Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press.
20.Karen Kukich. 1983. Design of a knowledge-based report generator. In Proceedings of ACL.
21.Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Workshop on Text Summarization Branches Out.
22.Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. Proceedings of EMNLP.
23.Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692.
24.Qingsong Ma, Ondřej Bojar, and Yvette Graham. 2018. Results of the wmt18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of WMT.
25.Qingsong Ma, Yvette Graham, Shugen Wang, and Qun Liu. 2017. Blend: a novel combined mt metric based on direct assessment–casict-dcu submission to wmt17 metrics task. In Proceedings of WMT.
26.Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the wmt19 metrics shared task: Segment-level and strong mt systems pose big challenges. In Proceedings of WMT.
27.Inderjeet Mani. 1999. Advances in automatic text summarization. MIT press.
28.Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In Proceedings of ACL.
29.Kathleen McKeown. 1992. Text generation. Cambridge University Press.
30.Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nlg. Proceedings of EMNLP.
31.Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL.
32.Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging nlp models. In Proceedings of ACL.
33.Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. Proceedings of ACL.
34.Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2018. Ruse: Regressor using sentence embeddings for automatic machine translation evaluation. In Proceedings of WMT.
35.Anastasia Shimorina, Claire Gardent, Shashi Narayan, and Laura Perez-Beltrachini. 2019. Webnlg challenge: Human evaluation results. Technical report.
36.Ronnie W Smith and D Richard Hipp. 1994. Spoken natural language dialog systems: A practical approach. Oxford University Press.
37.Milos Stanojevic and Khalil Sima’an. 2014. Beer: Better evaluation as ranking. In Proceedings of WMT.
38.Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of NIPS.
39.Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P Parikh. 2019. Sticking to the facts: Confident decoding for faithful data-to-text generation. arXiv:1910.08684.
40.Gaurav Singh Tomar, Thyago Duque, Oscar Täckström, Jakob Uszkoreit, and Dipanjan Das. 2017. Neural paraphrase identification of questions with noisy pretraining. Proceedings of the First Workshop on Subword and Character Level Models in NLP.
41.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS.
42.Oriol Vinyals and Quoc Le. 2015. A neural conversational model. Proceedings of ICML.
43.Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of ICLR.
44.John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards universal paraphrastic sentence embeddings. Proceedings of ICLR.
45.Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. Proceedings of NAACL HLT.
46.Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017. Challenges in data-to-document generation. Proceedings of EMNLP.
47.Stratos Xenouleas, Prodromos Malakasiotis, Marianna Apidianaki, and Ion Androutsopoulos. 2019. Sum-qe: a bert-based summary quality estimation model supplementary material. In Proceedings of EMNLP.
48.Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. Proceedings of ICLR.
49.Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. Proceedings of EMNLP.

Citation

MLA

Sellam, T., et al. “BLEURT: Learning Robust Metrics for Text Generation”. arXiv, 2020, http://arxiv.org/abs/2004.04696v5.

APA

Sellam, T., Das, D., & Parikh, A. P. (2020). BLEURT: Learning Robust Metrics for Text Generation. arXiv. http://arxiv.org/abs/2004.04696v5

Chicago

Sellam, T., D. Das, and A. P. Parikh. 2020. “BLEURT: Learning Robust Metrics for Text Generation”. arXiv. http://arxiv.org/abs/2004.04696v5.

Harvard

Sellam, T., Das, D. and Parikh, A.P. (2020) “BLEURT: Learning Robust Metrics for Text Generation”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2004.04696v5.

Vancouver

1. Sellam T, Das D, Parikh AP (2020) BLEURT: Learning Robust Metrics for Text Generation. arXiv

BibTeX

@article{sellam2020bleurt,
  title = {BLEURT: Learning Robust Metrics for Text Generation},
  author = {Sellam, Thibault and Das, Dipanjan and Parikh, Ankur P.},
  year = {2020},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2004.04696v5},
  eprint = {2004.04696}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

BLEURT: Learning Robust Metrics for Text Generation

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — BLEURT Model Architecture and Three-Stage Training Pipeline

Knowl 2 — Multi-Task Synthetic Pre-Training Objectives and Loss Formulation

Knowl 3 — Pre-Training Supervision Signals for Synthetic Metric Tuning

Knowl 4 — Generation of Synthetic Reference-Candidate Sentence Pairs

Knowl 5 — Supervised Fine-Tuning on Human Quality Judgments

Knowl 6 — Evaluation Performance on WMT Metrics Shared Tasks (2017–2019)

Knowl 7 — Robustness of BLEURT to Quality and Distribution Drift

Knowl 8 — Sample Efficiency and Few-Shot Adaptation on WebNLG

Knowl 9 — Impact and Ablation of Individual Pre-Training Signals

Knowl 10 — Pre-Training Efficiency and Convergence Schedule

References

Citation

BibTeX

Source Code

Access the Paper