Back

mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer

Linting XueNoah ConstantAdam RobertsMihir KaleRami Al-RfouAditya SiddhantAditya BaruaColin Raffel

article2020NAACL3,365 citations

Presents mT5, a multilingual text-to-text transformer trained across 101 languages that establishes state-of-the-art performance on cross-lingual benchmarks and provides a practical technique to prevent accidental translation in zero-shot generation.

Listen

The paper introduces mT5, a multilingual extension of the T5 text-to-text model, to overcome the English-only focus of most large language models. Roughly 80 percent of the global population does not speak English, so models trained solely on English data leave the majority of users without effective tools for core NLP tasks such as classification, named-entity recognition, and question answering.

The work set out to test whether the T5 recipe—unified text-to-text format, span-corruption pre-training, and large scale—could be applied directly to 101 languages while preserving strong performance. The authors built mC4, a cleaned Common Crawl corpus covering those languages, increased the vocabulary to 250,000 word pieces, and trained five model sizes ranging from 300 million to 13 billion parameters. They evaluated the models on the XTREME benchmark suite under zero-shot, translate-train, and in-language multitask conditions and introduced a lightweight remedy for “accidental translation,” in which a generative model produces part of its output in the wrong language.

The largest model, mT5-XXL, surpassed prior state-of-the-art results on five of the six XTREME tasks in the zero-shot setting and on all tasks when English training data plus machine translations were available. Performance gaps between zero-shot and fully supervised multilingual training narrowed sharply as model size grew, indicating that scale reduces the need for translated or gold target-language labels. The accidental-translation problem declined by more than 70 percent relative when a small fraction of the original mC4 pre-training task was mixed into fine-tuning. Ablations confirmed that a language-sampling exponent of 0.3 and a line-length filter each contributed measurable gains.

These outcomes show that a straightforward scaling of an existing English recipe can deliver competitive cross-lingual transfer without specialized parallel data or language-specific filtering. The public release of checkpoints and code therefore gives practitioners immediate access to a single model family that supports both classification and generation across many languages.

Further gains will likely require additional compute for even larger models, targeted data collection for the lowest-resource languages, and systematic study of inference-time constraints that keep generative outputs within legal spans. The main uncertainties are the high run-to-run variance observed on some tasks and the still-limited absolute performance on the scarcest languages.

Googlebilingualencoder-decodergenerative modelmultilingual language modelscross-lingual representation learningmultilingual pre-training task
arXiv: 2010.11934google-research/multilingual-t5
Cover for mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer

Abstract

The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent "accidental translation" in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.

Topics

bilingualencoder-decodergenerative modelnatural language processing

Keywords

multilingual language modelscross-lingual representation learningmultilingual pre-training taskmC4 CorpusmT5 modelstext classificationgenerative taskslanguage sampling

Table of Contents

  • 1 Introduction
  • 2 Background on T5 and C4
  • 3 mC4 and mT5
  • 3.1 mC4
  • 3.2 mT5
  • 3.3 Comparison to related models
  • 4 Experiments
  • 4.1 Results
  • 4.2 Ablation
  • 5 Zero-shot generation
  • 5.1 Illegal predictions
  • 5.2 Preventing accidental translation
  • 6 Conclusion

Knowls

  1. Knowl 1 — mT5 Model Architecture and Pre-training Configuration

    model/method

    mT5 is a massively multilingual variant of the Text-to-Text Transfer Transformer (T5) that maps arbitrary text inputs to text outputs using an encoder-decoder Transformer. It adopts the T5.1.1 architecture modifications, incorporating gated linear unit nonlinearities (GeGLU), scaling both the model hidden dimension dmodeld_{\text{model}} and feed-forward dimension dffd_{\text{ff}}, and omitting dropout during pre-training.

    mT5 is trained with a vocabulary of 250,000 subword tokens generated via SentencePiece using a character coverage of 0.99999 and byte-fallback enabled to uniquely represent rare characters. Models are pre-trained for 10610^6 steps with batches of 1,024 input sequences of length 1,024 (roughly 1 trillion tokens total). The pre-training objective is masked language modeling via span corruption, masking 15% of tokens with an average noise span length of 3 tokens. Optimization uses an inverse square-root learning rate schedule:

    η(n)=1max⁡(n,k)\eta(n) = \frac{1}{\sqrt{\max(n, k)}}

    where nn is the training step and k=104k = 10^4 is the number of warm-up steps.

    mT5 is provided in five parameter sizes:

    • mT5-Small: ≈300M\approx 300\text{M} parameters
    • mT5-Base: ≈580M\approx 580\text{M} parameters
    • mT5-Large: ≈1.2B\approx 1.2\text{B} parameters
    • mT5-XL: ≈3.7B\approx 3.7\text{B} parameters
    • mT5-XXL: ≈13B\approx 13\text{B} parameters

    The parameter counts are larger than their English T5 counterparts due to the larger 250,000-token embedding matrix.

  2. Knowl 2 — Multilingual C4 (mC4) Dataset Construction

    model/method

    Multilingual C4 (mC4) is a web-scraped natural language dataset covering 101 distinct languages (corresponding to 107 language-script varieties detected by Compact Language Detector v3, cld3), comprising approximately 6.6 billion pages and 6.3 trillion tokens extracted from 71 monthly Common Crawl dumps.

    The extraction and filtering pipeline consists of the following steps:

    1. Language Identification: Each web page is analyzed with cld3. Pages with an assigned language confidence score below 70% are discarded.
    2. Line Length Filtering: Rather than discarding lines lacking terminal English punctuation marks, a line length heuristic requires each retained page to contain at least three lines of text, each having 200 or more characters.
    3. Deduplication and Bad-Word Filtering: Duplicate lines across documents are removed, and pages containing profane or blocked terms are filtered out.
    4. Volume Thresholding: Pages are grouped by language code; only languages with at least 10,000 surviving web pages are retained, yielding 107 varieties (including six Romanized script variants such as Russian in Latin script).

  3. Knowl 3 — Temperature-Based Multilingual Data Sampling Probability

    equation

    During multilingual pre-training on the mC4 dataset, examples from different languages are sampled according to an exponentially smoothed probability distribution to prevent low-resource languages from underfitting and high-resource languages from overfitting:

    p(L)=∣L∣α∑L′∣L′∣αp(L) = \frac{|L|^\alpha}{\sum_{L'} |L'|^\alpha}

    where:

    • LL is a specific language in the corpus,
    • ∣L∣|L| is the total number of available examples (or pages) for language LL,
    • α∈(0,1]\alpha \in (0, 1] is the sampling exponent controlling the boosting of lower-resource languages.

    In mT5 pre-training, the smoothing parameter is set to α=0.3\alpha = 0.3. This setting provides a trade-off: higher values (such as α=0.7\alpha = 0.7) degrade performance on low-resource languages, while lower values (such as α=0.2\alpha = 0.2) compromise performance on high-resource languages.

  4. Knowl 4 — Accidental Translation and Illegal Predictions in Zero-Shot Generative Extractive QA

    definition

    In generative text-to-text models applied to extractive span-selection tasks (such as extractive question answering), an illegal prediction occurs when the model outputs a string that is not a verbatim substring of the input context. In cross-lingual zero-shot evaluation (fine-tuning on English target answers and testing on non-English inputs), illegal predictions fall into three primary classes:

    1. Normalization: The output text is semantically and lexically identical to the source context but differs in Unicode character representation (such as decomposed characters in Indic or Thai scripts or full-width punctuation marks). These can be resolved by Unicode NFKC normalization.
    2. Grammatical Adjustment: The model inflects or reformulates the target span to be morphologically and syntactically well-formed as an independent answer (for example, switching from instrumental case to nominative case in Russian, German, Turkish, or Arabic), even though the gold benchmark annotation is an exact grammatical substring.
    3. Accidental Translation: The model spontaneously translates part or all of the context span into the fine-tuning language (English). This manifests as full phrase translation, partial phrase translation, or intra-word code-switching where only a subword prefix is translated into English before reverting to the target script.

  5. Knowl 5 — Domain Preserving Training (DPT) for Mitigating Accidental Translation

    algorithm

    Domain Preserving Training (DPT) is a fine-tuning technique designed to prevent generative multilingual models from accidentally translating non-English outputs into English during zero-shot cross-lingual transfer. It mixes unsupervised pre-training data into downstream supervised fine-tuning.

    Input: Downstream supervised training set DtaskD_{\text{task}} (e.g., English QA pairs), Unlabeled multilingual corpus DmC4D_{\text{mC4}} across 101 languages
    Output: Fine-tuned text-to-text model parameters θ\theta
    Set mixing ratio to 1:100 (1 unsupervised mC4 example per 100 supervised task examples)
    Set multilingual sampling exponent α=0.1\alpha = 0.1 to sample languages near-uniformly
    for each fine-tuning batch do
        Sample labeled examples from DtaskD_{\text{task}}
        Sample unlabeled span-corruption examples from DmC4D_{\text{mC4}} according to p(L)∝∣L∣0.1p(L) \propto |L|^{0.1}
        Modify mC4 target sequences by removing all non-masked sentinel tokens
        Compute cross-entropy loss over combined batch
        Update θ\theta via gradient descent
    end for
    return θ\theta

    Applying DPT with a 1:100 mixing ratio and α=0.1\alpha = 0.1 reduces the zero-shot illegal prediction rate on XQuAD by more than 70% relative on smaller models (mT5-Small and mT5-Base) while lowering the overall error rate.

  6. Knowl 6 — Multilingual Benchmark Performance Across Model Scales (XTREME)

    empirical result

    When evaluated across diverse cross-lingual benchmarks from XTREME, mT5 scales consistently from Small to XXL, achieving state-of-the-art results on classification and question-answering tasks without requiring task-specific parallel data or intermediate cross-lingual objectives.

    Model XNLI PAWS-X WikiAnn NER XQuAD MLQA TyDiQA-GoldP
    Acc. Acc. F1 F1 / EM F1 / EM F1 / EM
    Cross-lingual zero-shot transfer (fine-tuned on English only)
    mBERT 65.4 81.9 62.2 64.5 / 49.4 61.4 / 44.2 59.7 / 43.9
    XLM-R (Large) 79.2 86.4 65.4 76.6 / 60.8 71.6 / 53.2 65.1 / 45.0
    RemBERT 80.8 87.5 70.1 79.6 / 64.0 73.1 / 55.0 77.0 / 63.0
    mT5-Small 67.5 82.4 50.5 58.1 / 42.5 54.6 / 37.1 35.2 / 23.2
    mT5-Base 75.4 86.4 55.7 67.0 / 49.0 64.6 / 45.0 57.2 / 41.2
    mT5-Large 81.1 88.9 58.5 77.8 / 61.5 71.2 / 51.7 69.9 / 52.2
    mT5-XL 82.9 89.6 65.5 79.5 / 63.6 73.5 / 54.5 75.9 / 59.4
    mT5-XXL 85.0 90.0 69.2 82.5 / 66.8 76.0 / 57.4 80.8 / 65.9
    Translate-train (fine-tuned on English plus machine translations)
    XLM-R (Large) 82.6 90.4 – 80.2 / 65.9 72.8 / 54.3 66.5 / 47.7
    FILTER 83.9 91.4 – 82.4 / 68.0 76.2 / 57.7 68.3 / 50.9
    mT5-XXL 87.8 91.5 – 85.2 / 71.3 76.9 / 58.3 82.8 / 68.8

    mT5-XXL outperforms prior models on all zero-shot and translate-train classification and question-answering benchmarks, matching or approaching specialized models on structured NER prediction.

  7. Knowl 7 — Scaling Eliminates Capacity Sharing Penalties on Monolingual Tasks

    empirical result

    Massively multilingual models typically experience a performance penalty on monolingual English benchmarks relative to dedicated monolingual models of equal size due to parameter allocation across over 100 languages. Increasing model scale closes this gap on the SQuAD reading comprehension benchmark (evaluated in F1 / Exact Match scores):

    Model Scale T5 (Monolingual English) mT5 (Multilingual)
    Small 87.2 / 79.1 84.7 / 76.4
    Base 92.1 / 85.4 89.6 / 83.8
    Large 93.8 / 86.7 93.0 / 87.0
    XL 95.0 / 88.5 94.5 / 88.9
    XXL 96.2 / 91.3 95.6 / 90.4

    While mT5-Small and mT5-Base lag behind monolingual T5 by 2.5 and 2.5 F1 points respectively, mT5-XL and mT5-XXL reduce the deficit to 0.5 and 0.6 F1 points, demonstrating that cross-lingual interference diminishes significantly at large model capacity.

  8. Knowl 8 — Model Scale Closes the Gap Between Zero-Shot Transfer and In-Language Multitask Training

    empirical result

    On the TyDi QA GoldP reading comprehension benchmark, the performance disparity between fine-tuning exclusively on English (zero-shot), fine-tuning on English plus synthetic machine translations (translate-train), and fine-tuning on gold target-language annotations (in-language multitask) decreases markedly as parameter count increases.

    • At the Small scale (≈300M\approx 300\text{M} parameters), zero-shot transfer yields 34.9 F1 / 23.4 EM, lagging behind translate-train (48.2 F1) and in-language multitask training (73.0 F1).
    • At the XXL scale (≈13B\approx 13\text{B} parameters), zero-shot transfer reaches 81.0 F1 / 65.6 EM, approaching translate-train (82.8 F1 / 68.8 EM) and substantially closing the gap to in-language multitask gold training (88.5 F1 / 79.1 EM).

    This indicates that sufficiently large generative multilingual models diminish the necessity of generating synthetic translations or annotating target-language data for cross-lingual transfer.

  9. Knowl 9 — Ablation Analysis of mT5 Pre-training Choices on XNLI and XQuAD

    empirical result

    Ablation experiments conducted on the mT5-Large configuration demonstrate the impact of pre-training dataset filtering, sequence length, language sampling rate, and regularization on zero-shot XNLI accuracy (and zero-shot XQuAD F1/EM):

    • Baseline mT5-Large (mC4 data, line length filter, α=0.3\alpha=0.3, sequence length 1024, noise span length 3, pre-training dropout 0): 81.1% XNLI accuracy (77.8 / 61.5 XQuAD F1/EM).
    • No line length filter: 79.1% (−-2.0%), demonstrating that eliminating short, low-quality web scrapes is critical.
    • Adding Wikipedia data (103 languages): 80.3% (−-0.8%), showing no benefit over mC4 alone.
    • Pre-training Dropout 0.1: 77.6% (−-3.5%), confirming that dropout during unsupervised pre-training impairs representation quality.
    • Average Span Length 10 (vs. 3): 78.6% (−-2.5%).
    • Sequence Length 512 (vs. 1024): 80.5% (−-0.6%).
    • Sampling Exponent α=0.7\alpha = 0.7: 80.7% (improves high-resource languages like Russian from 81.5% to 82.8%, but drops low-resource languages like Swahili from 75.4% to 70.6%).
    • Sampling Exponent α=0.2\alpha = 0.2: 80.7% (boosts tail languages like Urdu from 73.5% to 73.9%, but lowers average accuracy across all languages).

Coverage note — Omitted the exhaustive per-language result breakdown tables for individual tasks (Tables 6, 7, 8, 9, 10, 11, 12, 13, 14) which contain dozens of individual language scores; their aggregate findings and cross-lingual trends are fully captured in the benchmark and ablation knowls.

References

  1. 1.Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint arXiv:1907.05019.
  2. 2.Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.
  3. 3.Diedre Carmo, Marcos Piau, Israel Campiotti, Rodrigo Nogueira, and Roberto Lotufo. 2020. PTT5: Pre-training and validating the t5 model on brazilian portuguese data. arXiv preprint arXiv:2008.09144.
  4. 4.Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2020. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training. arXiv preprint arXiv:2007.07834.
  5. 5.Hyung Won Chung, Thibault Févry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2020. Rethinking embedding coupling in pre-trained language models. arXiv preprint arXiv:2010.12821.
  6. 6.Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470.
  7. 7.Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.
  8. 8.Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems, volume 32, pages 7059–7069.
  9. 9.Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  10. 10.David Crystal. 2008. Two thousand million? English today, 24(1):3–6.
  11. 11.Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. BERTje: A dutch BERT model. arXiv preprint arXiv:1912.09582.
  12. 12.Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020. RobBERT: a dutch RoBERTa-based language model. arXiv preprint arXiv:2001.06286.
  13. 13.Jacob Devlin. 2018. Multilingual BERT README. https://github.com/google-research/bert/blob/master/multilingual.md.
  14. 14.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  15. 15.Yuwei Fang, Shuohang Wang, Zhe Gan, Siqi Sun, and Jingjing Liu. 2020. FILTER: An enhanced fusion method for cross-lingual language understanding. arXiv preprint arXiv:2009.05166.
  16. 16.Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. Association for Computational Linguistics.
  17. 17.Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics.
  18. 18.Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multitask benchmark for evaluating cross-lingual generalization. arXiv preprint arXiv:2003.11080.
  19. 19.Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282.
  20. 20.Mihir Kale. 2020. Text-to-text pre-training for data-to-text tasks. arXiv preprint arXiv:2005.10433.
  21. 21.Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Unifying question answering and text classification via span extraction. arXiv preprint arXiv:1904.09286.
  22. 22.Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UnifiedQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1896–1907, Online. Association for Computational Linguistics.
  23. 23.Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational Linguistics.
  24. 24.Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  25. 25.Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabbé, Laurent Besacier, and Didier Schwab. 2020. FlauBERT: Unsupervised language model pre-training for French. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 2479–2490, Marseille, France. European Language Resources Association.
  26. 26.Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. 2020a. Pre-training via paraphrasing. arXiv preprint arXiv:2006.15020.
  27. 27.Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020b. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics.
  28. 28.Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019. MLQA: Evaluating cross-lingual extractive question answering. arXiv preprint arXiv:1910.07475.
  29. 29.Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020a. Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.08210.
  30. 30.Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
  31. 31.Zihan Liu, Genta Indra Winata, Andrea Madotto, and Pascale Fung. 2020b. Exploring fine-tuning techniques for pre-trained cross-lingual models via continual learning. arXiv preprint arXiv:2004.14218.
  32. 32.Fuli Luo, Wei Wang, Jiahao Liu, Yijia Liu, Bin Bi, Songfang Huang, Fei Huang, and Luo Si. 2020. Veco: Variable encoder-decoder pre-training for cross-lingual understanding and generation. arXiv preprint arXiv:2010.16046.
  33. 33.Martin Malmsten, Love Börjeson, and Chris Haffenden. 2020. Playing with words at the national library of sweden–making a swedish BERT. arXiv preprint arXiv:2007.01658.
  34. 34.Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203–7219, Online. Association for Computational Linguistics.
  35. 35.Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730.
  36. 36.Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020. WT5?! Training text-to-text models to explain their predictions. arXiv preprint arXiv:2004.14546.
  37. 37.Dat Quoc Nguyen and Anh Tuan Nguyen. 2020. PhoBERT: Pre-trained language models for Vietnamese. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1037–1042, Online. Association for Computational Linguistics.
  38. 38.Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pre-trained sequence-to-sequence model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708–718, Online. Association for Computational Linguistics.
  39. 39.Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. Association for Computational Linguistics.
  40. 40.Jason Phang, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann, Iacer Calixto, and Samuel R Bowman. 2020. English intermediate-task training improves zero-shot cross-lingual transfer too. arXiv preprint arXiv:2005.13013.
  41. 41.Marco Polignano, Pierpaolo Basile, Marco de Gemmis, Giovanni Semeraro, and Valerio Basile. 2019. AlBERTo: Italian BERT language understanding model for NLP challenging tasks based on tweets. In CLiC-it.
  42. 42.Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
  43. 43.Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  44. 44.Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online. Association for Computational Linguistics.
  45. 45.Sebastian Ruder, Matthew E. Peters, Swabha Swayamdipta, and Thomas Wolf. 2019. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pages 15–18, Minneapolis, Minnesota. Association for Computational Linguistics.
  46. 46.Noam Shazeer. 2020. GLU variants improve transformer. arXiv preprint arXiv:2002.05202.
  47. 47.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998–6008.
  48. 48.Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4003–4012, Marseille, France. European Language Resources Association.
  49. 49.Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3687–3692, Hong Kong, China. Association for Computational Linguistics.

Citation

MLA
Xue, L., et al. “mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer”. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 483–98, https://doi.org/10.18653/v1/2021.naacl-main.41.
APA
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou’, R., Siddhant, A., Barua, A., & Raffel, C. (2021). mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 483–498. https://doi.org/10.18653/v1/2021.naacl-main.41
Chicago
Xue, L., N. Constant, A. Roberts, et al. 2021. “mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer”. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 483–98. https://doi.org/10.18653/v1/2021.naacl-main.41.
Harvard
Xue, L. et al. (2021) “mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer”, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pp. 483–498. Available at: https://doi.org/10.18653/v1/2021.naacl-main.41.
Vancouver
1. Xue L, Constant N, Roberts A, Kale M, Al-Rfou’ R, Siddhant A, Barua A, Raffel C (2021) mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pp 483–498

BibTeX

@inproceedings{xue-etal-2021-mt5,
    title = "m{T}5: A Massively Multilingual Pre-trained Text-to-Text Transformer",
    author = "Xue, Linting  and
      Constant, Noah  and
      Roberts, Adam  and
      Kale, Mihir  and
      Al-Rfou, Rami  and
      Siddhant, Aditya  and
      Barua, Aditya  and
      Raffel, Colin",
    editor = "Toutanova, Kristina  and
      Rumshisky, Anna  and
      Zettlemoyer, Luke  and
      Hakkani-Tur, Dilek  and
      Beltagy, Iz  and
      Bethard, Steven  and
      Cotterell, Ryan  and
      Chakraborty, Tanmoy  and
      Zhou, Yichao",
    booktitle = "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
    month = jun,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.naacl-main.41/",
    doi = "10.18653/v1/2021.naacl-main.41",
    pages = "483--498"
}
Metadata:ACL Anthology

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/