Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pre-trained Models

Taido Purason $^{1}$, Pavel Chizhov $^{2}$, Ivan P. Yamshchikov $^{2}$, Mark Fishel $^{1}$
$^{1}$ Institute of Computer Science, University of Tartu
$^{2}$ CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt
[email protected]

Abstract

Tokenizer adaptation plays an important role in adapting pre-trained language models to new domains or languages. In this work, we address two complementary aspects of this process: vocabulary extension and pruning. The common approach to extension trains a new tokenizer on domain-specific text and appends the tokens that do not overlap with the existing vocabulary, which often results in many tokens that are unreachable or never used. We propose continued BPE training that extends a pre-trained tokenizer by continuing the BPE merge learning process on new data. Experiments across multiple languages and model families show that this approach improves tokenization efficiency and leads to better utilization of added vocabulary. We also introduce leaf-based vocabulary pruning, which removes redundant tokens while preserving model quality. Together, these methods provide practical tools for controlled vocabulary modification, which we release as an open-source toolkit.

Executive Summary: Adapting large language models to new languages or domains often relies on continued pre-training, yet the tokenizer that breaks text into tokens frequently remains inefficient for underrepresented languages. This leads to longer sequences, higher compute costs, and slower performance. The standard way to add domain- or language-specific tokens—training a fresh tokenizer on target data and simply appending non-overlapping entries—introduces many tokens that never participate in the actual tokenization process and therefore deliver little benefit.

The work set out to develop and test more effective ways to extend and prune tokenizer vocabularies while preserving compatibility with existing models and implementations. Researchers introduced continued BPE training, which resumes the original tokenizer’s merge-learning process directly on target-domain text, and leaf-based vocabulary pruning, which removes low-value tokens by following the structure of the merge graph rather than frequency counts alone. They evaluated these methods across four model families and 70 languages using large multilingual corpora, measuring both compression efficiency and downstream task performance after limited continued pre-training.

Continued BPE training produced 3–9.6 % better average compression than the naive approach, with gains exceeding 20 % for some languages and no unreachable tokens introduced. Leaf-based pruning safely removed up to 62.5 % of vocabulary entries without degrading downstream accuracy and outperformed simpler frequency-based or last-N strategies. When the improved tokenizer was used in bilingual English–Estonian continued pre-training, total training tokens dropped by roughly 26 % while task scores remained comparable to those obtained with the unmodified tokenizer.

These results matter because tokenization efficiency directly affects training time, inference cost, and memory footprint. Vocabulary extension improves handling of new languages, while pruning keeps model size—and especially the large embedding layer—under control. The methods preserve full compatibility with standard model frameworks and do not require changes to inference code.

Organizations adapting models to new languages or domains should adopt continued BPE training for extensions and leaf-based pruning for size reduction. The authors released an open-source toolkit that implements both techniques for common tokenizer formats. Further work is warranted on broader multilingual settings and on quantifying the precise trade-offs between vocabulary size and full-scale model performance. Experiments were conducted primarily in bilingual scenarios and focused on BPE tokenizers, so results should be confirmed in more complex multilingual and non-BPE contexts before large-scale deployment.

1. Introduction

Section Summary: When adapting large language models to new languages or domains, researchers often update the model's tokenizer alongside its weights to handle inefficient encoding that inflates sequence lengths and computing costs. This work introduces a more effective extension method that continues training the original byte-pair encoding process on target data, rather than adding tokens from a separately trained tokenizer, and pairs it with a structure-preserving pruning approach that removes unused tokens without harming performance or creating unreachable vocabulary items. The authors also release an open toolkit to support these modifications for existing models.

When adapting large language models (LLMs) to new domains or languages, continued pre-training has become a widely used strategy. However, effective adaptation depends not only on updating model weights but also the tokenizer. Most LLMs rely on byte-pair encoding [BPE, 1, 2] tokenizers, and in practice their effectiveness varies substantially across languages. In particular, inefficient tokenization for underrepresented languages often leads to longer sequences and higher computational costs in LLMs [3]. One way to address this is to extend the vocabulary with domain-specific tokens, which improves compression and reduces sequence lengths, albeit at the cost of increasing the model size due to a larger vocabulary. Pruning infrequent or irrelevant tokens can address this trade-off by reducing vocabulary size. In this work, we focus on tokenizer modification of pre-trained models using those two steps: pruning and extension.

$**Figure 1:** Change in text compression (*bytes per token* $\uparrow$) of text using the Llama 3 tokenizer extended with Estonian tokens using naive tokenizer extension and continued BPE training (ours). We also compare to a tokenizer trained from scratch on Estonian data.$

$**Figure 2:** Change in text compression (*bytes per token* $\uparrow$) for vocabulary pruning methods on Estonian-English pruning (Llama-3). Note that *Freq (leaf)* and *Merge-based* method results overlap.$

**Figure 3:** The comparison of vocabulary extension methods: previously common approach of training a tokenizer and using its vocabulary for extending an existing tokenizer (left) and continuing BPE training on an existing tokenizer proposed by us (right).

Extending an existing vocabulary is a widely used step in adapting pre-trained LLMs to new languages [4, 5, 6, 7, 8, 9, 10, 11]. In previous works, vocabulary extension typically involved training a new tokenizer on in-domain data and appending the tokens that were previously not present to the original tokenizer (see Figure 3). We show that this approach introduces useless tokens that do not participate in merges, resulting in lower compression compared to having tokens that are useful in tokenization. This happens because BPE strictly follows the merge sequence, e.g., when a word information is originally tokenized as inform + ation, adding new tokens infor and mation is useless, even though they are new to the vocabulary. In this work, we propose a more effective alternative: continuing to train the original BPE tokenizer on domain-specific data. We show that for the same number of added tokens, this approach yields better compression (see Figure 1). While we focus on LLMs, our method can be applied to other pre-trained models that use BPE tokenizers.

The other side of tokenizer adaptation, besides extension, is pruning. Thus, we also study the impacts of different tokenizer pruning strategies and introduce a non-invasive leaf-based pruning method that trims the vocabulary in a structure-preserving manner. This avoids creating unreachable tokens and yields higher tokenization efficiency (see Figure 2). By using token frequencies in a target corpus, akin to tokenizer training, we can preserve domain- and language-specific tokens without increasing sequence lengths in relevant domains while pruning irrelevant tokens. Our pruning method can also be used on its own in the context of model compression to reduce the vocabulary size, thereby substantially improving the model's efficiency: for instance, Gemma 3 [12] has 270 million parameters, 170 million of which are used for vocabulary embeddings [13].

In summary, our contributions are threefold:

First, we introduce a new method for vocabulary extension based on continued BPE training, and provide a detailed analysis showing that it yields more efficient tokenization than the widely used approach of adding tokens from an independently trained domain-specific tokenizer.
Second, we develop improved vocabulary pruning strategies that do not change the tokenizer implementation and systematically analyze their effectiveness.
Finally, we release an open-source toolkit^1 that supports modification of both Byte-level and SentencePiece BPE tokenizers for pre-trained models through Hugging Face transformers [HF, 14].

2. Related work

Section Summary: Research shows that tokenization quality strongly influences model performance, with language-specific approaches often outperforming multilingual ones and better compression generally leading to stronger results. Prior work has explored extending tokenizers by training new ones on targeted data and adding tokens through various merging or matching techniques, as well as pruning low-value tokens using frequency or morphological criteria, though these methods can introduce incompatibilities or require workarounds during use. Other studies have replaced tokenizers entirely or focused on updating model embeddings for new vocabularies, but such changes often demand substantial retraining.

A growing body of work shows that tokenization quality is tied to model performance, highlighting its importance. Language-specific tokenizers have been shown to provide advantages over multilingual ones ([15]), and more efficient compression is associated with better downstream results ([16]). Recent evidence also suggests that differences in tokenization quality may contribute to the performance gap observed between agglutinative and fusional languages ([17]).

2.1 Tokenizer extension

Several works adopted the strategy of training a new tokenizer on domain- or language-specific data and adding the resulting non-overlapping tokens to the original vocabulary [4, 5, 6, 7, 8, 9, 10, 11], which we refer to as the naive approach. MEDVOC-LLM [18] provides another way to add new tokens, such as those proposed by a newly trained, domain-specific tokenizer. Specifically, each new token is tokenized with the original tokenizer and then recreated by merging the constituent tokens from left to right, adding intermediate merges and scaffold tokens. AdaptBPE [19] integrates new tokens via longest string matching before tokenization, but this alters the tokenization function itself.

In contrast to the aforementioned methods, rather than specifying the desired tokens in advance and trying to integrate them, continued BPE training (ours) directly continues the BPE training process from the pre-trained tokenizer on in-domain data. This ensures that all added tokens and merges are fully compatible and optimal under the BPE objective, without introducing any unreachable tokens, thereby solving both the problem of identifying the tokens to add and their optimal integration.

2.2 Tokenizer pruning

In addition to extension, we introduce improved pruning techniques that surpass naive frequency-based methods [20, 6], allowing for more extensive and targeted pruning without compromising compression or downstream performance. Prior pruning methods, such as BPE-Trimming [21], target low-frequency tokens and split them during inference. We extend this work by introducing leaf-based pruning and trimming the tokenizer itself so that the inference remains identical to the basic BPE. BPE-Knockout [22] removes tokens that have little value in terms of morphology and introduces tuple merges to compensate for their absence. PickyBPE [23] and Scaffold-BPE [24] represent a different line of work of refining the vocabulary during tokenizer training by removing the intermediate, "scaffolding" tokens. While our pruning method also removes tokens, it does so to eliminate unused tokens in application to a narrower set of languages or domains. Our approach therefore modifies only the vocabulary and merge list while leaving the tokenization function unchanged, maintaining full compatibility with the original implementation and enabling further extensions.

2.3 Replacing the tokenizer

Instead of pruning and extending, previous works have also successfully replaced the previous tokenizer with a new one entirely [25]. However, [26] show that replacing or significantly altering the tokenizer requires extensive continued pre-training to avoid performance degradation, and [4] find that the larger the vocabulary extension, the longer training is needed. We leave the question of what is the best strategy—modifying or replacing the tokenizer—to future research.

2.4 Modification of embeddings

When the tokenizer vocabulary is modified, the model embeddings must be updated accordingly. [4] and [5] find that Fast Vocabulary Transfer [FVT, 27], which copies embeddings for overlapping tokens and initializes new ones by averaging those of their decompositions under the original tokenizer (see Appendix E), performs competitively among initialization strategies. While alternative methods exist [28, 25], our work focuses on selecting which tokens to add or prune at the tokenizer level rather than on embedding initialization.

3. Methodology

Section Summary: The methodology describes two main techniques for adapting a tokenizer's vocabulary to new domains. For extension, one approach adds tokens from a freshly trained in-domain tokenizer along with newly generated merge rules, while an improved method called continued BPE training reapplies the byte-pair encoding algorithm to in-domain text by counting pair frequencies on segments already tokenized by the original model and respecting each tokenizer's specific rules, such as pre-tokenization or script constraints. For controlled reduction, pruning removes low-priority tokens and their merges from the vocabulary using either ID- or frequency-based selection, refined by a leaf-based algorithm that iteratively identifies and eliminates only those tokens that do not participate in further merges, thereby avoiding unreachable entries.

3.1 Vocabulary extension

Naive extension

It is common to extend a tokenizer with new tokens by training a new tokenizer with in-domain data and adding the non-overlapping tokens in the order of the trained vocabulary. As HF transformers relies on merge lists, it is also necessary to create merges leading to the added tokens. We generate the merges leading to the new tokens based on the token priorities, as opposed to simply merging the two tokenizers' merge lists, since we found the prior provides better results (see Appendix B.5). We refer to this method as the naive approach.

Continued tokenizer training

As an improvement, we propose continued tokenizer training to improve tokenization efficiency (see Figure 3). We define continued BPE training as reapplying the BPE tokenizer training algorithm to in-domain data, using token pair frequencies of the text tokenized with the existing tokenizer to create the new merges. During merge creation, we ensure that each resulting token is valid according to the training implementation of the underlying tokenizer.

For Byte-level BPE models [BBPE 29], this involves first applying the pre-tokenizer to segment the text, then tokenizing the resulting segments to count token pair frequencies within the pre-tokens.

For SentencePiece BPE models [30], we tokenize the text and follow SentencePiece's merging rules by respecting constraints such as keeping different scripts separate and only allowing the space symbol at the beginning of a token. If necessary, we also extend the vocabulary with missing characters to ensure coverage before finding extension merges and ensure that the training data is Unicode NFKC normalized.

In both cases, we then apply the standard BPE algorithm for tokenizer training using pair frequencies from the tokenized texts and creating merges until the desired number of new tokens is reached. For extending the original tokenizer we simply append the new tokens to the vocabulary and the new merges to the merge list in the order they were created. Similar to BPE tokenizer training, this method is language-agnostic and only requires texts in the target language.

3.2 Vocabulary Pruning

To allow for controlled reduction of vocabulary size, we also implement vocabulary pruning. This process involves removing tokens and the merges that lead to them, freeing space for vocabulary extension. To determine the tokens that should be pruned first, we try several approaches, based either on token ID (Last N) or on token frequency in a representative text sample. When implemented naively, both approaches might lead to unreachable tokens: IDs do not always reflect the order of merges, as in Llama 3.2 [31], and intermediate tokens tend to have lower frequencies than their downstream full-word tokens [23].

Addressing this issue, we implemented a leaf-based pruning algorithm, which iteratively collects the pruning order of tokens that are leaves in the BPE merge graph, i.e., they do not participate in merges and thus do not produce any downstream tokens. In Algorithm 1, we show pseudocode for frequency-based pruning of leaf tokens. First, atomic and leaf tokens are selected using the merge list. Leaf tokens are used to populate the priority queue, while atomic tokens cannot be removed and should never be added to the queue. After this, on each iteration, the lowest-priority element is taken from the queue and split into its merging sub-tokens. For each of the subtokens, we check the number of downstream merges it participates in, and, if there are no downstream merges left, this token is also considered a leaf and added to the priority queue for removal. The process ends when the queue runs out of leaves, which is the point when all non-atomic tokens are split.

We also introduce a merge-based pruning algorithm, which yields results similar to leaf-based frequency pruning. As its performance closely matches that of leaf-based pruning, we provide a detailed description in Appendix A.

**Input:**
  $\textcolor{#4DB6AC}{\texttt{vocab}}$: mapping "token → vocabulary index"
  $\textcolor{#4DB6AC}{\texttt{unreachable}}$: set of unreachable tokens
  $\textcolor{#4DB6AC}{\texttt{merges}}$: list of BPE merges
  $\textcolor{#4DB6AC}{\texttt{counts}}$: mapping "token → frequency in the target corpus"
**Output:**
$\textcolor{#4DB6AC}{\texttt{prune\_order}}$: list of tokens to prune in the order of pruning.
$\textcolor{#7268D8}{\textbf{initialize}}$ $\textcolor{#4DB6AC}{\texttt{atomics}}$ ← set of tokens not reachable by merges
$\textcolor{#7268D8}{\textbf{initialize}}$ $\textcolor{#4DB6AC}{\texttt{leaves}}$ ← set of merged tokens not participating in merges + $\textcolor{#4DB6AC}{\texttt{unreachable}}$
$\textcolor{#7268D8}{\textbf{initialize}}$ $\textcolor{#4DB6AC}{\texttt{downstream\_merges}}$ ← mapping "token → number of merges with it"
$\textcolor{#7268D8}{\textbf{initialize}}$ $\textcolor{#4DB6AC}{\texttt{token\_splits}}$ ← mapping "token → merge leading to it"
$\textcolor{#7268D8}{\textbf{initialize}}$ $\textcolor{#4DB6AC}{\texttt{prune\_order}}$ ← []
$\textcolor{#7268D8}{\textbf{initialize}}$ $\textcolor{#4DB6AC}{\texttt{queue}}$ ← heap priority queue "leaf ← (frequency, index)"
**while** $\textcolor{#4DB6AC}{\texttt{queue}}$:
  $\textcolor{#4DB6AC}{\texttt{token}}$ ← $\textcolor{#4DB6AC}{\texttt{queue}}$.popmin()
  $\textcolor{#4DB6AC}{\texttt{pruning\_order}}$.append($\textcolor{#4DB6AC}{\texttt{token}}$)
  (t₁, t₂) ← $\textcolor{#4DB6AC}{\texttt{token\_splits}}$ [$\textcolor{#4DB6AC}{\texttt{token}}$]
  $\textcolor{#7268D8}{\textbf{update}}$ t₁, t₂ in $\textcolor{#4DB6AC}{\texttt{queue}}$ by $\textcolor{#4DB6AC}{\texttt{queue}}$ [$\textcolor{#4DB6AC}{\texttt{token}}$]
  **for** t ∈ (t₁, t₂):
    $\textcolor{#4DB6AC}{\texttt{downstream\_merges}}$ [ t ] -= 1
    **if** ( $\textcolor{#4DB6AC}{\texttt{downstream\_merges}}$ [ t ] == 0) and (t ∉ $\textcolor{#4DB6AC}{\texttt{atomics}}$ ):
      add t to $\textcolor{#4DB6AC}{\texttt{leaves}}$ and $\textcolor{#4DB6AC}{\texttt{queue}}$
**return** $\textcolor{#4DB6AC}{\texttt{pruning\_order}}$

3.3 Detecting unreachable tokens

We extend the method of [32] for identifying unreachable tokens to all vocabulary items (including those not decodable to UTF-8) and provide a formal definition. We define the Self-Tokenization Test (STT) as the number of tokens unreachable through merges:

$ STT = \sum_{t \in \mathcal{V}} \mathbf{1}\left[tokenize(t) \neq [t] \right] $

The main idea is that if tokenizing a vocabulary token with the same tokenizer[^2] does not reproduce that token, then the token cannot be formed through merge operations[^3].

[^2]: In practice, this means that we test the raw string corresponding to the token, bypassing pre-processing, and disabling merge skipping (ignore_merges).

[^3]: It may still be produced through merge skipping.

Rationale.

Let a BPE tokenizer be defined by a vocabulary $\mathcal{V}$ and a deterministic sequence of local merges, applied without merge skipping. For a token $t \in \mathcal{V}$, if $\text{tokenize}(t) \neq [t]$, then $t$ is unreachable by merges. If $\text{tokenize}(t) = [t]$, then $t$ could be reachable.

Suppose $t$ appears in the tokenized output of some pre-tokenized input $\mathcal{I}$. Then there must exist a substring of $\mathcal{I}$ equal to the literal form of $t$ that is merged into $t$. Because merges are local and merge skipping is disabled, the same sequence of merges must also apply when tokenizing the isolated string $t$. Therefore, if $\text{tokenize}(t) \neq [t]$, no input string can ever produce $t$, and $t$ is unreachable. However, if $\text{tokenize}(t) = [t]$, then $t$ is in principle reachable by merges assuming the pre-tokenizer produces $t$ as a substring to some input.

Merge Skipping.

If a pre-token output by a pre-tokenizer fully matches a vocabulary token, the merging process is skipped, and the token is produced immediately when merge skipping is enabled (ignore_merges in HF). When applying this method, merge skipping is disabled because otherwise any existing vocabulary token trivially returns $[t]$, making the test meaningless. Some tokenizers (e.g., Llama 3) rely on this mechanism to produce certain tokens that are reachable only via merge skipping. While our method offers a fast check of merge-based reachability, its interpretation must account for whether tokens are produced via merge skipping in practice.

4. Experimental setup

Section Summary: The experimental setup used training data from the Fineweb and Fineweb-2 corpora with fixed budgets of 100 million characters across 70 languages and 1 billion for Estonian, plus a larger 24-billion-character bilingual mix of English and Estonian for a targeted case study on language adaptation. Evaluation measured tokenization efficiency via bytes per token and Rényi efficiency on the FLORES-200 benchmark, while downstream performance was tested on translation and reasoning tasks in both languages using standard benchmarks. The study compared four existing models with different base tokenizers and conducted training experiments by adapting the Llama-3.2 1B and 3B models with modified vocabularies.

4.1 Datasets

We use Fineweb-2 ([33], ODC-By license, multilingual) and Fineweb ([34], ODC-By license, English) as the training datasets and set a training budget of 100M characters for the comparison across 70 languages (see Table 12 in Appendix F) and 1B characters for a more targeted evaluation for Estonian. We observe that increasing the amount of training data beyond this threshold does not yield significant improvements in tokenization efficiency during extension (see Appendix B.1).

We frame the extrinsic evaluation as a case study on Estonian, using English-Estonian datamix for both tokenizer and LM training. Our setup reflects a common bilingual LLM adaptation scenario: extending a model to a new language while preserving its English capabilities. To this end, we allocated a total budget of 24B characters, equally divided between Estonian and English. The corpus was then tokenized using the different tokenizers under evaluation, ensuring that the underlying data remained constant across experiments involving changes to the tokenizer.

4.2 Evaluation

We evaluate compression on the FLORES-200 [35] devtest split using bytes per token ($BPT=\frac{\text{UTF-8 bytes}}{\text{tokens}}$) as the evaluation metric (higher is better). For comparison of different methods, we report the relative difference $\Delta_{rel}BPT = \frac{BPT_{ours}}{BPT_{naive}}-1$, which essentially states how many additional tokens the naive method produces compared to ours, as the number of bytes remains constant ($\frac{tokens_{naive}}{tokens_{ours}}-1$). We additionally report Rényi efficiency ([36], higher is better), which has been shown to correlate with downstream performance.

For downstream evaluation in Estonian and English, we evaluate on FLORES-200 [37] (ET $\leftrightarrow$ EN), Winogrande [38], Winogrande-ET [39], XCOPA [40], SIB200 [41], and Belebele [42] using lm-evaluation-harness [43]. We evaluate FLORES-200 with COMET [Unbabel/wmt22-comet-da, 44, 45]. See Appendix D for more details.

4.3 Models

We look at four models Llama-3 [31], Llama-2 [46], Qwen-2.5 [47] and, Mistral Nemo^4. While Llama-3, Qwen-2.5, and Mistral Nemo use the BBPE tokenizer, while Llama-2 uses a SentencePiece BPE tokenizer. We use the tokenizers through Hugging Face transformers implementation [14]. For experiments involving model training, we use Llama-3.2 1B and 3B and modify its embeddings after vocabulary modification with FVT [27].

5. Results

Section Summary: Continuing tokenizer training produces up to 9.6 percent better average compression across 70 languages than the naive method of adding tokens from a separately trained tokenizer, with most languages showing clear gains and no meaningful loss in English performance. These efficiency improvements reduce the number of tokens needed to process text, lowering computational costs during training and inference. While raw Rényi efficiency scores look slightly lower, the continued-training approach still outperforms the naive method once tokenizers are compared at similar compression levels.

5.1 Tokenizer evaluation

\begin{tabular}{@l@cc@{ }c@ccc@cccc@}\toprule
 {} & &\multicolumn{2}{c}{Compression $\uparrow$} &\textbf{} &\multicolumn{2}{c}{\textbf{Rényi effic.} $\uparrow$} &\textbf{} &\multicolumn{2}{c}{\textbf{Unreach. added} $\downarrow$} \\\cmidrule{3-4}\cmidrule{6-7}\cmidrule{9-10}
\multicolumn{2}{c}{\textbf{Model}} &$\Delta_{rel}$ &\textbf{WR} &\textbf{} &$\Delta_{rel}$ &\textbf{WR} &\textbf{} &\textbf{ours} &\textbf{naive} \\\midrule
\multicolumn{10}{l}{\cellcolor[HTML]{f3f3f3}\textbf{Llama-3}} \\
&+1000 &\cellcolor[HTML]{f1ecfd}3.3\% &88.6\% & &\cellcolor[HTML]{fefbfb}-0.1\% &18.6\% & &0.0\% &\cellcolor[HTML]{fcede9}4.5\% \\
&+2000 &\cellcolor[HTML]{efeafc}3.6\% &92.9\% & &\cellcolor[HTML]{fdf4f2}-0.6\% &4.3\% & &0.0\% &\cellcolor[HTML]{fbe8e3}5.7\% \\
&+4000 &\cellcolor[HTML]{eee9fc}3.8\% &98.6\% & &\cellcolor[HTML]{fceeeb}-1.1\% &5.7\% & &0.0\% &\cellcolor[HTML]{fae3dd}7.0\% \\
&+8000 &\cellcolor[HTML]{eee9fc}3.7\% &100.0\% & &\cellcolor[HTML]{fbebe7}-1.4\% &1.4\% & &0.0\% &\cellcolor[HTML]{f9ded7}8.2\% \\
&+16000 &\cellcolor[HTML]{f0ecfd}3.3\% &98.6\% & &\cellcolor[HTML]{fbebe7}-1.3\% &1.4\% & &0.0\% &\cellcolor[HTML]{f8d9d1}9.3\% \\
&+32000 &\cellcolor[HTML]{f3f0fd}2.7\% &92.9\% & &\cellcolor[HTML]{fbeeeb}-1.1\% &7.1\% & &0.0\% &\cellcolor[HTML]{f8d6cd}10.2\% \\\midrule
\multicolumn{10}{l}{\cellcolor[HTML]{f3f3f3}\textbf{Qwen-2.5}} \\
&+1000 &\cellcolor[HTML]{ece6fc}4.3\% &98.6\% & &0.1\% &18.6\% & &0.0\% &\cellcolor[HTML]{fceeea}4.3\% \\
&+2000 &\cellcolor[HTML]{e6defb}5.6\% &97.1\% & &\cellcolor[HTML]{fcf0ed}-1.0\% &7.1\% & &0.0\% &\cellcolor[HTML]{fbe8e4}5.6\% \\
&+4000 &\cellcolor[HTML]{e0d6fa}6.9\% &100.0\% & &\cellcolor[HTML]{fae6e1}-1.8\% &2.9\% & &0.0\% &\cellcolor[HTML]{fae2dc}7.1\% \\
&+8000 &\cellcolor[HTML]{dacff9}8.1\% &98.6\% & &\cellcolor[HTML]{f8dcd5}-2.5\% &1.4\% & &0.0\% &\cellcolor[HTML]{f9ddd6}8.4\% \\
&+16000 &\cellcolor[HTML]{d6caf8}9.0\% &97.1\% & &\cellcolor[HTML]{f7d6ce}-3.0\% &1.4\% & &0.0\% &\cellcolor[HTML]{f8d8d0}9.6\% \\
&+32000 &\cellcolor[HTML]{d3c6f7}9.6\% &94.3\% & &\cellcolor[HTML]{f7d4cb}-3.2\% &8.6\% & &0.0\% &\cellcolor[HTML]{f7d4cb}10.5\% \\\midrule
\multicolumn{10}{l}{\cellcolor[HTML]{f3f3f3}\textbf{Mistral Nemo}} \\
&+1000 &\cellcolor[HTML]{f2edfd}3.0\% &97.1\% & &\cellcolor[HTML]{fdf6f4}-0.5\% &12.9\% & &0.0\% &\cellcolor[HTML]{fdf0ed}3.8\% \\
&+2000 &\cellcolor[HTML]{eee9fc}3.7\% &94.3\% & &\cellcolor[HTML]{fcf2f0}-0.8\% &5.7\% & &0.0\% &\cellcolor[HTML]{fcebe7}4.9\% \\
&+4000 &\cellcolor[HTML]{ece6fc}4.2\% &98.6\% & &\cellcolor[HTML]{fcefec}-1.0\% &4.3\% & &0.0\% &\cellcolor[HTML]{fbe6e0}6.3\% \\
&+8000 &\cellcolor[HTML]{ebe5fc}4.5\% &98.6\% & &\cellcolor[HTML]{fbeeea}-1.1\% &4.3\% & &0.0\% &\cellcolor[HTML]{fae1da}7.5\% \\
&+16000 &\cellcolor[HTML]{ebe4fc}4.6\% &100.0\% & &\cellcolor[HTML]{fcefeb}-1.1\% &4.3\% & &0.0\% &\cellcolor[HTML]{f9dcd4}8.7\% \\
&+32000 &\cellcolor[HTML]{ece6fc}4.3\% &98.6\% & &\cellcolor[HTML]{fcf1ef}-0.9\% &4.3\% & &0.0\% &\cellcolor[HTML]{f8d8cf}9.7\% \\\midrule
\multicolumn{10}{l}{\cellcolor[HTML]{f3f3f3}\textbf{Llama-2 (SentencePiece)}} \\
&+1000 &\cellcolor[HTML]{eae4fc}4.6\% &95.7\% & &\cellcolor[HTML]{fef9f8}-0.2\% &17.1\% & &0.0\% &\cellcolor[HTML]{fdf1ee}3.6\% \\
&+2000 &\cellcolor[HTML]{e8e1fb}5.2\% &95.7\% & &\cellcolor[HTML]{fcefeb}-1.1\% &8.6\% & &0.0\% &\cellcolor[HTML]{fcece8}4.8\% \\
&+4000 &\cellcolor[HTML]{e4dbfa}6.1\% &90.0\% & &\cellcolor[HTML]{fae6e1}-1.8\% &5.7\% & &0.0\% &\cellcolor[HTML]{fbe7e2}6.0\% \\
&+8000 &\cellcolor[HTML]{e0d6fa}6.9\% &84.3\% & &\cellcolor[HTML]{f9dfd8}-2.3\% &7.1\% & &0.0\% &\cellcolor[HTML]{fae3dd}7.0\% \\
&+16000 &\cellcolor[HTML]{ddd3f9}7.4\% &72.9\% & &\cellcolor[HTML]{f8dad2}-2.7\% &17.1\% & &0.0\% &\cellcolor[HTML]{fadfd9}7.8\% \\
&+32000 &\cellcolor[HTML]{dcd2f9}7.7\% &74.3\% & &\cellcolor[HTML]{f7d8d0}-2.8\% &21.4\% & &0.0\% &\cellcolor[HTML]{f9ddd6}8.4\% \\
\bottomrule
\end{tabular}

By continuing tokenizer training (ours) instead of extending a tokenizer from a new language-specific tokenizer (naive method), we achieve up to 9.6% higher tokenization efficiency on average, with a vast majority (72.9%–100% depending on the tokenizer and extension size) of the 70 languages achieving higher compression (see Table 1). We do not observe any notable negative effects of our method on compression, while the increase can exceed 20% compared to the naive method (see Appendix G.2 for full results). Furthermore, there is also almost no effect on English tokenization after the target-language extension (see Table 6 in Appendix B.3). Higher target-language compression means we can process the same text with fewer tokens during training and inference, consuming less computational resources.

When looking at Rényi efficiency (higher is better), there is a deterioration when comparing continued BPE training to naive extension. On a closer look, even though Rényi efficiency decreases for the model with the same number of added tokens, when looking at extended tokenizers with roughly equal compression, our method outperforms the naive method on average (see Figure 4).

$**Figure 4:** Llama-3 tokenizer Rényi efficiency $\uparrow$ vs compression (bytes per token) $\uparrow$ averaged over 70 languages. The point labels indicate the number of added tokens.$

While our method is language-agnostic, we observe that the improvement over the naive method depends on the base tokenizer, language, and the number of added tokens. If we examine the average increase in compression of our method over the naive method, we see that it is more effective for languages with Cyrillic and Latin scripts (see Table 2).

\begin{tabular}{@l@{ }rccccc@}\toprule
 {} & &\multicolumn{4}{c}{$\Delta_{rel}$ Compression $\uparrow$} \\\cmidrule{3-6}
\textbf{Script}&($n$)&\textbf{Llama-3} &\textbf{Qwen-2.5} &\textbf{Mistral} &\textbf{Llama-2} \\\midrule
Latn &(35) &\cellcolor[HTML]{ede8fc}4.7\% &\cellcolor[HTML]{d9cdf8}9.8\% &\cellcolor[HTML]{f3effd}3.3\% &\cellcolor[HTML]{d8ccf8}10.1\% \\
Cyrl &(9) &\cellcolor[HTML]{ece6fc}5.1\% &\cellcolor[HTML]{d3c6f7}11.2\% &\cellcolor[HTML]{eae4fc}5.4\% &\cellcolor[HTML]{ddd3f9}8.7\% \\
Arab &(4) &\cellcolor[HTML]{fdfcff}0.7\% &\cellcolor[HTML]{f1ecfd}3.8\% &\cellcolor[HTML]{fefeff}0.4\% &\cellcolor[HTML]{fffeff}0.2\% \\
Jpan &(1) &\cellcolor[HTML]{faf8ff}1.4\% &\cellcolor[HTML]{fbfaff}1.1\% &\cellcolor[HTML]{faf8fe}1.4\% &\cellcolor[HTML]{fcf1ef}-1.0\% \\
Hani &(1) &\cellcolor[HTML]{fefefe}-0.0\% &\cellcolor[HTML]{fffeff}0.2\% &0.0\% &\cellcolor[HTML]{fefdfc}-0.1\% \\
Other &(20) &\cellcolor[HTML]{fbf9ff}1.2\% &\cellcolor[HTML]{f7f4fe}2.3\% &\cellcolor[HTML]{e8e1fb}5.9\% &\cellcolor[HTML]{fefdff}0.5\% \\
\bottomrule
\end{tabular}

Why is continued BPE training more effective?

In both vocabulary extension methods, each newly added token has merges leading to it from the tokens already in the tokenizer. However, this does not guarantee that these tokens will actually be produced during inference. To quantify this, we introduce the Self-Tokenization Test (see Section 3.3), which calculates tokens unreachable through merges. Extending from an independently trained tokenizer (naive) produces more unreachable tokens compared to the continued tokenizer training, which does not produce any (see Table 1). This means that naive vocabulary extension has added tokens that will never be used in the merge process.

\begin{tabular}{lcccccccc}\toprule
&\multicolumn{3}{c}{\textbf{Llama-3}} & &\multicolumn{3}{c}{\textbf{Qwen-2.5}} \\\cmidrule{2-4}\cmidrule{6-8}
\textbf{Ext. size} &\textbf{ours} &\textbf{naive} &\textbf{WR} & &\textbf{ours} &\textbf{naive} &\textbf{WR} \\\midrule
+1000 &0.0\% &\cellcolor[HTML]{fdf1ee}4.0\% &85.7\% & &0.0\% &\cellcolor[HTML]{fcefeb}8.3\% &88.6\% \\
+2000 &0.1\% &\cellcolor[HTML]{fdf0ed}4.3\% &92.9\% & &0.0\% &\cellcolor[HTML]{fcece8}9.7\% &98.6\% \\
+4000 &0.1\% &\cellcolor[HTML]{fcefeb}4.6\% &98.6\% & &0.1\% &\cellcolor[HTML]{fbe9e4}11.0\% &100.0\% \\
+8000 &\cellcolor[HTML]{fffefe}0.5\% &\cellcolor[HTML]{fcede9}5.1\% &98.6\% & &0.4\% &\cellcolor[HTML]{fbe6e0}12.6\% &100.0\% \\
+16000 &\cellcolor[HTML]{fef8f7}2.1\% &\cellcolor[HTML]{fbe6e1}6.9\% &100.0\% & &\cellcolor[HTML]{fffbfb}2.0\% &\cellcolor[HTML]{fae0d9}15.3\% &100.0\% \\
+32000 &\cellcolor[HTML]{fbe6e0}7.1\% &\cellcolor[HTML]{f7d4cb}11.8\% &100.0\% & &\cellcolor[HTML]{fdf1ee}7.0\% &\cellcolor[HTML]{f7d4cb}20.9\% &100.0\% \\
\bottomrule
\end{tabular}

It is important to note that the Llama-3 and Mistral Nemo use merge skipping by default, so some of the tokens unreachable through merges will still be produced in practice. To quantify the effect of this, we disable merge skipping for Llama-3 and Mistral Nemo and find that for the naive method, the compression drops 1.2%–5.8% and 0.5%–4.1% respectively on average across 70 languages depending on the extension size (see Table 7 in Appendix B.4). For our method, the difference is less than 0.1% on average. This might explain why the difference between our method and the naive method is smaller for Llama-3 and Mistral Nemo, that use merge skipping, compared to Qwen-2.5 and Llama-2, which do not.

::: {caption="Table 4: Interaction of pruning and extension (Llama-3). Bytes-per-token (BPT) on Flores and unreachable tokens with different pruning and extension methods. In each experiment, we prune the tokenizer by $N_D$elta tokens using 50%–50% Estonian-English data mix, and then extend it back to the original size using Estonian-only data. *L indicates that leaf-based pruning was used."}

:::

In addition to calculating the number of tokens we can not reach through merges, we also tokenized the held-out target-language dataset using extended Llama-3 and Qwen-2.5 tokenizers to see which tokens are not produced in practice. On average, across 70 languages, our method yields a 4%–13.9% improvement in the utilization of the added token vocabulary in practice (see Table 3). Furthermore, we observed more low-frequency tokens after naive extension compared to continued BPE training when visualizing Estonian token counts (see Appendix B.2).

Pruning.

When pruning and then extending the tokenizers[^5] (see Table 4), we see that in terms of text compression, frequency-aware methods lead to better progress in added language (Estonian) and slower degradation in the base tokenizer language (English). In addition, leaf-based methods perform comparably or better than analogous naive implementations, as they account for the BPE structure. This is more evident in the number of unreachable tokens, where naive frequency-based pruning leads to a dramatic increase in unreachable tokens, breaking the BPE merge paths. Our results conclude that the leaf-based frequency pruning is the most effective approach. Merge-based pruning, our other proposed algorithm, performs comparably to the leaf-based pruning.

[^5]: It is also possible to first extend and then prune; however, we did not observe any meaningful difference in performance depending on the order (see Appendix B.6).

We provide a comparison of bilingual pruning without extension for nine languages in Appendix G.1, similarly demonstrating the benefits of leaf- and merge-based pruning methods.

\begin{tabular}{@lcccccccccccccc@c@c@}\toprule
&\textbf{Compr.} & &\multicolumn{2}{c}{\textbf{FLORES} \tiny{(COMET)}} &\textbf{} &\multicolumn{2}{c}{\textbf{Winogrande} \tiny{(acc)}} &\textbf{} &\multicolumn{2}{c}{\textbf{Belebele} \tiny{(acc)}} &\textbf{} &\multicolumn{2}{c}{\textbf{SIB200} \tiny{(acc)}} &\textbf{} &\textbf{XCOPA} \tiny{(acc)} \\\cmidrule{4-5}\cmidrule{7-8}\cmidrule{10-11}\cmidrule{13-14}
\textbf{Model} &\textbf{ET \tiny{(EN)}} &\textbf{Toks.} &\textbf{EN-ET} &\textbf{ET-EN} &\textbf{} &\textbf{ ET} &\textbf{EN} &\textbf{} &\textbf{ET} &\textbf{EN} &\textbf{} &\textbf{ET} &\textbf{EN} &\textbf{} &\textbf{ET} \\\midrule
\cellcolor[HTML]{f3f3f3}\textbf{Llama-3.2-1B} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} \\
No training &2.64 \tiny{(4.86)} &0 &0.432 \tiny{± 0.008} &0.701 \tiny{± 0.007} & &50.0 \tiny{± 2.3} &61.2 \tiny{± 2.7} & &28.6 \tiny{± 3.0} &35.7 \tiny{± 3.1} & &71.6 \tiny{± 6.2} &77.0 \tiny{± 5.8} & &51.0 \tiny{± 4.4} \\
\textbf{Continued pretraining:} &\textbf{} &\textbf{} & & & & & & & & & & & & & \\
- default tokenizer &2.64 \tiny{(4.86)} &7.2B &0.788 \tiny{± 0.008} &0.803 \tiny{± 0.005} & &58.1 \tiny{± 2.3} &59.1 \tiny{± 2.7} & &28.0 \tiny{± 2.9} &26.9 \tiny{± 2.9} & &77.9 \tiny{± 5.7} &74.5 \tiny{± 6.0} & &63.6 \tiny{± 4.2} \\
- prune+ext (16k, naive) &4.21 \tiny{(4.86)} &5.4B &0.790 \tiny{± 0.008} &0.809 \tiny{± 0.005} & &57.4 \tiny{± 2.3} &58.8 \tiny{± 2.7} & &25.6 \tiny{± 2.9} &25.7 \tiny{± 2.9} & &73.0 \tiny{± 6.1} &73.0 \tiny{± 6.1} & &62.6 \tiny{± 4.2} \\
- prune+ext (16k, ours) &4.46 \tiny{(4.86)} &5.3B &0.796 \tiny{± 0.007} &0.803 \tiny{± 0.005} & &58.3 \tiny{± 2.3} &58.6 \tiny{± 2.7} & &27.3 \tiny{± 2.9} &27.1 \tiny{± 2.9} & &76.5 \tiny{± 5.8} &73.0 \tiny{± 6.1} & &63.0 \tiny{± 4.2} \\
 {} & & & & & & & & & & & & & & & \\
\cellcolor[HTML]{f3f3f3}\textbf{Llama-3.2-3B} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} &\cellcolor[HTML]{f3f3f3}\textbf{} \\
No training &2.64 \tiny{(4.86)} &0 &0.635 \tiny{± 0.010} &0.796 \tiny{± 0.006} & &53.0 \tiny{± 2.3} &69.1 \tiny{± 2.5} & &44.1 \tiny{± 3.2} &74.2 \tiny{± 2.9} & &74.5 \tiny{± 6.0} &76.0 \tiny{± 5.9} & &55.4 \tiny{± 4.4} \\
\textbf{Continued pretraining:} &\textbf{} &\textbf{} & & & & & & & & & & & & & \\
- default tokenizer &2.64 \tiny{(4.86)} &7.2B &0.835 \tiny{± 0.006} &0.830 \tiny{± 0.005} & &65.1 \tiny{± 2.2} &68.2 \tiny{± 2.6} & &52.9 \tiny{± 3.3} &64.7 \tiny{± 3.1} & &80.9 \tiny{± 5.4} &77.0 \tiny{± 5.8} & &70.4 \tiny{± 4.0} \\
- prune+ext (16k, naive) &4.21 \tiny{(4.86)} &5.4B &0.818 \tiny{± 0.008} &0.831 \tiny{± 0.005} & &63.2 \tiny{± 2.2} &68.2 \tiny{± 2.6} & &51.1 \tiny{± 3.3} &66.0 \tiny{± 3.1} & &81.4 \tiny{± 5.4} &78.9 \tiny{± 5.6} & &69.8 \tiny{± 4.0} \\
- prune+ext (16k, ours) &4.46 \tiny{(4.86)} &5.3B &0.822 \tiny{± 0.007} &0.829 \tiny{± 0.005} & &63.1 \tiny{± 2.3} &68.4 \tiny{± 2.6} & &48.7 \tiny{± 3.3} &65.0 \tiny{± 3.1} & &78.4 \tiny{± 5.7} &77.9 \tiny{± 5.7} & &71.8 \tiny{± 3.9} \\
\bottomrule
\end{tabular}

5.2 Downstream evaluation

5.2.1 Continued pre-training

Downstream task results for the continually pre-trained Llama-3.2 1B and 3B models in Table 5 show no substantial difference between the naive and continued tokenizer training approaches. In these experiments, we first applied leaf-based frequency pruning and then extended the vocabulary to maintain a constant size. When comparing the original and modified tokenizers, the 1B model performs similarly in both cases, while the 3B model exhibits slightly lower scores on EN-ET machine translation and Belebele (ET) for the modified tokenizers, while having similar results on other benchmarks. Nevertheless, modifying the tokenizer reduced total training time by 26% compared to the base tokenizer, owing to improved compression for Estonian (see Table 9). When looking at the embeddings, we observe signs of undertrained tokens for the naive method after continued pre-training compared to our method. A detailed analysis is provided in Appendix B.2.

5.2.2 Model Compression via Pruning

If we consider the scenario of using a model for a specific language or a domain, multilingual tokenizers typically include tokens for many languages that may not be required. We investigate how many of those tokens we can remove without affecting the downstream performance. Focusing on Estonian and English, we remove tokens from the vocabulary and the embeddings while aiming to preserve performance in these two languages.

**Figure 5:** Llama-3.1-8B vocabulary pruned with different methods and evaluated on Estonian and English downstream tasks. The dataset for training the pruner consisted of 50-50 Estonian-English mix. The highlighted area shows the 95% confidence interval acquired through bootstrap resampling. Note that *Freq (leaf)* and *Merge-based* method results overlap.

With our pruning methods, up to 62.5% (80k/128k) of the vocabulary tokens can be removed without any noticeable deterioration in downstream performance (see Figure 5). When comparing pruning strategies, leaf-based frequency pruning and merge-based pruning consistently allow for more extensive token removal than last-N pruning and naive frequency pruning. The latter two methods exhibit performance degradation earlier, whereas our approaches maintain robustness under higher pruning ratios. We also observe that machine translation (generative) is more sensitive to vocabulary pruning than the discriminative tasks, where the amount of tokens that can be removed without loss in performance is even higher than 62.5%. We also observe similar results for German-English pruning (see Figure 11 in Appendix G.1).

6. Discussion

Section Summary: The researchers show that their method of continuing to train an existing tokenizer produces better results than the usual quick extension approach, while also preserving the original structure and avoiding any wasted tokens that the standard method often creates. They further introduce a pruning technique that trims the tokenizer in a structured way, either alone or together with extension, to shrink the overall model size. This pruning can also factor in word frequencies to drop the least useful items, allowing a balanced sample of languages or domains to be represented even if they were not part of the original extension.

Our method of continued tokenizer training yields better intrinsic tokenizer performance than the commonly utilized naive extension method. More importantly, unlike naive extension, it does not harm the BPE structure and leaves no unreachable tokens. Unreachable tokens introduced by the naive method are only partially addressed by merge skipping, as merge skipping uses only the tokens that constitute complete pre-tokens.

Leaf-based pruning enables structure-aware tokenizer trimming, which can be utilized in conjunction with the tokenizer extension or independently to reduce model size, a factor that is now largely dependent on embedding parameters, particularly for small models. Furthermore, introducing frequencies into this pruning approach makes it possible to remove the vocabulary needed the least. For example, it is possible to combine a representative sample of all languages or domains required in the fine-tuned model, not only the ones directly participating in the tokenizer extension.

7. Conclusion

Section Summary: The researchers introduced techniques to expand or shrink the word lists used by pre-trained language models so they can handle new topics or languages more efficiently. One method retrains the model's word-splitting system directly on specialized data instead of simply adding new words, while a companion pruning step safely removes redundant entries without hurting accuracy. The authors also released an open-source toolkit that implements these changes and works with existing model libraries.

We presented methods for controlled vocabulary modification of pre-trained language models through improved tokenizer extension and pruning. Our proposed continued BPE training extends an existing tokenizer directly on domain-specific data, avoiding the inefficiencies of appending tokens from an independently trained tokenizer. Experiments across multiple languages and model families demonstrate that this approach enhances tokenization efficiency and more effectively utilizes the extended vocabulary. In parallel, our leaf-based pruning algorithm enables safe vocabulary reduction by removing redundant tokens without degrading model performance. Together, these techniques provide a practical framework for adapting tokenizers to new domains and languages while maintaining model compactness and quality. To help facilitate further research on vocabulary adaptation, we release an open-source toolkit implementing our methods, while preserving compatibility with the Hugging Face format.

Limitations

Section Summary: The study’s experiments were limited to a bilingual setup involving one added language and focused only on BPE tokenizers in large language models, leaving more complex multilingual cases, other segmentation methods, and different model architectures unexplored. The authors also chose not to investigate how best to optimize overall model performance after tokenizer changes or to determine when full replacement would be preferable. Most real-world task evaluations were further restricted to a single language and model, which may limit how far the findings generalize.

While our method demonstrates improved tokenization efficiency through continued BPE training, several limitations remain.

Our experiments were conducted in a bilingual setting, where a single language was added to an existing tokenizer. Although this setup is typical in low-resource adaptation, more complex multilingual scenarios remain unexplored and warrant future investigation.

The current approach is also restricted to BPE-based tokenizers, which are still the predominant choice in large language models. Extending the method to other segmentation algorithms could provide additional insights.

Although our evaluation focuses on large language models, the method itself is not inherently limited to them. It could be applied to any pre-trained model that relies on a BPE tokenizer. However, other architectures, such as encoder-decoder NMT models or encoder-only language models, were not included in our experiments.

We deliberately limit the scope of this work to tokenizer-level modification and do not explore how to achieve the best model-level downstream performance. This question likely depends on multiple interacting factors, such as dataset size, model capacity, and embedding initialization strategies, and thus requires a more extensive investigation. Consequently, we do not address when a tokenizer should be replaced entirely or how much tokenizer extension is optimal from the perspective of downstream performance.

Finally, the extrinsic evaluation on downstream tasks was, for the most part, limited to a single language and model, which may affect the generalizability of our findings to other settings.

8. Acknowledgments

Section Summary: The research received funding from Estonia's Ministry of Education and Research through a national program aimed at advancing Estonian language technology, specifically under project EKTB104. The authors also credit a German high-performance computing center for supplying essential computing resources. This support came via a partnership between a university in Erlangen and an artificial intelligence center at another technical institution.

This work was supported by the National Program for Estonian Language Technology Program (project EKTB104) funded by the Estonian Ministry of Education and Research. The authors also acknowledge the HPC resource allocation by Erlangen National High-Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) in a joint project with the Center for Artificial Intelligence (CAIRO), THWS.

Appendix

Section Summary: The appendix describes an alternative merge-based pruning technique that tracks token frequencies across the full BPE merging process, allowing similar leaf-node removal as the main frequency method while confirming the latter's focus in the paper. It also presents extra results showing that continued BPE training outperforms naive extension even with far less data, produces fewer undertrained tokens as measured by embedding shifts, leaves English tokenization nearly unchanged, and benefits substantially from merge skipping for better compression.

A. Merge-based pruning

**Input:**
  `vocab`: mapping from token → vocabulary index
  `tok(counts)`: token frequencies in tokenized corpus
  `merge(counts)`: frequencies of merges performed when tokenizing the corpus
**Output:**
`prune(order)`: list of tokens to prune in the order of pruning.
**initialize** `counts` ← `tok(counts)`
**for** (t₁, t₂), n in `merge(counts)`:
  `counts`[t₁] ← `counts`[t₁] + n
  `counts`[t₂] ← `counts`[t₂] + n
**return** **sort** tokens by ascending `counts`, break ties by descending length of the token and finally `vocab` index

As an alternative to the frequency based pruning with leaf ordering, we also developed the merge-based pruning (see Algorithm 2), which achieves similar results. The main idea is to count the token frequency throuout the mergining process. So, in practice this means counting the final token frequency and any time it appears in a merge, creating another token. Consequently, a vocabulary token will have a count of $n$, while any token it was merged from will have a count of at least $n$. This follows from the fact that standard BPE defines a single deterministic merge tree for each token, and tokens appearing earlier in such trees often participate in multiple merge trees. This means that if we sort by that count, breaking ties by token length (when a token and its predecessor in the merge tree have equal count) and token index, only the leaf nodes will be removed.

Since the result is very similar we decided to focus on the leaf-frequency pruning in the main part of the paper.

B. Additional results

B.1 Training dataset size

The effect of the training set diminishes at higher sizes. We see minimal difference between 1B and 10B character budget in scores when extending the tokenizer (see Figure 6). Extending a tokenizer using continued training with 10M character training set yields a bigger effect than extending from a new tokenizer (naive) using 10B character training set – the choice of extension method is more important than the training budget at this scale.

**Figure 6:** Extension training budget in characters vs compression of Estonian text when adding 16, 000 Estonian tokens to the Llama-3 tokenizer.

B.2 Analysis of undertrained tokens

Frequency distribution of the added tokens is visualized in Figure 7. We observe that the naive method yields more low-frequency tokens than our continued BPE training method.

**Figure 7:** Token frequency distribution in the added tokens ($n=8000$) in Llama-3 tokenizer on the held-out set of Estonian Fineweb-2 ($10000$ documents).

**Figure 8:** Comparison of input/output embeddings (tied) of added tokens at FVT initialization and after continued pre-training (L2 distance between them and the difference in L2 norms). We extend Llama-3.2-3B with Estonian tokens and continually pre-train on Estonian-English mix, comparing naive tokenizer extension and continued BPE training.

To better understand how this affects the learning of new tokens, we analyze the changes in their embeddings during continued pre-training. Since the added tokens are initialized with FVT and updated during training, undertraining is not always immediately visible. To investigate this, we measure both the L2 distance of embeddings from their initialized values and the change in their L2 norms. Figure 8 shows that the naive extension method produces noticeably more undertrained tokens, reflected in smaller L2 changes and a stronger weight-decay effect that drives their embeddings toward zero (visible as a longer tail of negative L2 norm differences). This interpretation is further supported by Table 3, which shows lower vocabulary utilization for the naive method.

B.3 Extension effect on English

Table 6 demonstrates that the target-language extension both with continued BPE training (ours) and the naive method has almost no effect on the tokenization of English sentences.

\begin{tabular}{@lccccc@}\toprule
\textbf{} &\multicolumn{2}{c}{Lev. dist. $\downarrow$} &\multicolumn{2}{c}{EM (\%) $\uparrow$} \\\cmidrule(lr){2-3}\cmidrule(lr){4-5}
\textbf{Ext. size} &\textbf{ours} &\textbf{naive} &\textbf{ours} &\textbf{naive} \\\midrule
1000 &0.001 \tiny{<0.010} &0.001 \tiny{<0.010} &100.0 &99.9 \\
2000 &0.002 \tiny{<0.012} &0.002 \tiny{<0.013} &99.9 &99.9 \\
4000 &0.003 \tiny{<0.022} &0.004 \tiny{<0.022} &99.9 &99.8 \\
8000 &0.009 \tiny{<0.036} &0.010 \tiny{<0.036} &99.6 &99.6 \\
16000 &0.023 \tiny{<0.106} &0.024 \tiny{<0.098} &99.1 &99.0 \\
32000 &0.057\tiny{ <0.198} &0.057 \tiny{<0.220} &97.7 &97.6 \\
\bottomrule
\end{tabular}

B.4 The effect of merge skipping

We note that merge skipping contributes significantly to the compression. We look at models that use merge skipping by default (Llama-3 and Mistral Nemo) and disable merge skipping to see how much the performance drops. In Table 7, see that for continued BPE training, disabling merge skipping has no significant effect, while for the naive method, the drop in tokenization efficiency is notable. This is because merge skipping allows reaching tokens that are unreachable by merges if those tokens are equal to a pre-token, bypassing merges. As we previously noted, the naive method produces many of those unreachable tokens.

\begin{tabular}{lcccccc}\toprule
&\multicolumn{2}{c}{\textbf{Llama-3}} & &\multicolumn{2}{c}{\textbf{Mistral Nemo}} \\\cmidrule{2-3}\cmidrule{5-6}
&\textbf{ours} &\textbf{naive} &\textbf{} &\textbf{ours} &\textbf{naive} \\\midrule
+1000 &\cellcolor[HTML]{fefefe}-0.1\% &\cellcolor[HTML]{fdf5f3}-1.2\% &\textbf{} &0.0\% &\cellcolor[HTML]{fefbfa}-0.5\% \\
+2000 &\cellcolor[HTML]{fefefe}-0.1\% &\cellcolor[HTML]{fcf0ed}-1.9\% &\textbf{} &0.0\% &\cellcolor[HTML]{fdf8f6}-0.9\% \\
+4000 &\cellcolor[HTML]{fefefe}-0.1\% &\cellcolor[HTML]{fbeae5}-2.8\% &\textbf{} &0.0\% &\cellcolor[HTML]{fcf3f0}-1.6\% \\
+8000 &\cellcolor[HTML]{fefefe}-0.1\% &\cellcolor[HTML]{f9e2dc}-3.8\% &\textbf{} &0.0\% &\cellcolor[HTML]{fbede9}-2.4\% \\
+16000 &\cellcolor[HTML]{fefefe}-0.1\% &\cellcolor[HTML]{f8dad3}-4.9\% &\textbf{} &0.0\% &\cellcolor[HTML]{fae6e1}-3.3\% \\
+32000 &\cellcolor[HTML]{fefefe}-0.1\% &\cellcolor[HTML]{f7d4cb}-5.8\% &\textbf{} &0.0\% &\cellcolor[HTML]{f9e0d9}-4.1\% \\
\bottomrule
\end{tabular}

B.5 Naive extension methods

There are multiple ways of adding tokens from an auxiliary independently trained tokenizer to an existing tokenizer (naive method).

$**Figure 9:** Comparison of **compression** (*bytes per tokens* $\uparrow$) of Estonian (target language) and English, and **unreachable tokens** $\downarrow$ given the number of added tokens for various methods of adding new tokens for Llama-3.$

For tokenizers that do not define merge lists explicitly (e.g. SentencePiece), it is common to just append the vocabulary [5, 11, 4]. For Hugging Face (HF) transformers tokenizer implementations, it is more complicated because merges are performed according to a merge list, so in addition to adding tokens, we also need to handle the merges. Some works have just appended the merge list of a new tokenizer [6, 4], however, we show later that this is not optimal.

We took the approach of generating merges for the added vocabulary (regen) by checking which tokens can create the added token and then ordering the merge rules by the priority/index. Our implementation is based on the way that HF transformers converts SentencePiece tokenizers to their format, which requires creating merge lists. This is also similar to what happens when tokens are added to a SentencePiece tokenizer, since there are no merge lists that are merged together – the added tokens are produced through merging according to their priority.

In Figure 9, we observe that the approach of generating merge rules for the added tokens (regen) works better than appending merge rules (merge). We also compared against using the add_tokens method of HF transformers, however we find it has the lowest performance.

There are also works that have prepended merges [6]. When used with the regen method, it does result in higher scores, however we see that it affects the English tokenization and results in more unreachable tokens, so we do not use it.

Thus we decide to use the regen method, since we try to use the strongest baseline to compare our method against.

B.6 Order of pruning/extension

Table 8 shows that the difference between pruning and extending (in that order) vs extending and pruning is negligible in terms of compression.

\begin{tabular}{ccccc}\toprule
&\multicolumn{2}{c}{\textbf{Prune-Extend}} &\multicolumn{2}{c}{\textbf{Extend-prune}} \\\cmidrule(r){2-3}\cmidrule(l){4-5}
$N_\Delta$ &ET &EN &ET &EN \\\midrule
16000 &4.441 &4.863 &4.441 &4.863 \\
32000 &4.441 &4.863 &4.441 &4.863 \\
64000 &4.440&4.861 &4.439 &4.861 \\
96000 &4.402 &4.671 &4.401 &4.671 \\
\bottomrule
\end{tabular}

C. Continued pre-training details

We provide details for the modified Llama-3 tokenizers in the continued pertaining experiments in Table 9.

The hyperparamters for continued pre-training using Hugging Face transformers [14] are in Table 10. The training was conducted on the LUMI supercomputer using 8 nodes, each consisting of 4 AMD MI250x GPUs (acting as 8 compute units). The continued pre-training experiments took 2311 GPU-hours (180–612 GPU-hours per experiment).

\begin{tabular}{@lrrccccc@}\toprule
 {} & & &\multicolumn{2}{c}{\textbf{Compr.}} &\textbf{CPT} & \\\cmidrule{4-5}
\textbf{Tokenizer} &\textbf{|vocab|} &\textbf{unreach} &\textbf{ET} &\textbf{EN} &\textbf{tokens} &\textbf{\% ET} \\\midrule
default &128k &588 &2.64 &4.86 &7.2B &63.9\% \\
prune+ext (16k) &128k &0 &4.46 &4.86 &5.3B &50.6\% \\
prune+ext (16k, naive) &128k &2033 &4.21 &4.86 &5.4B &52.0\% \\
\bottomrule
\end{tabular}

\begin{tabular}{lrr}\toprule
Hyperparameter &Value \\\midrule
Learning rate &4e-4 (1B) / 1e-4 (3B)\\
Optimizer &AdamW \\
Adam $\epsilon$ &1e-8 \\
Adam $\beta_1, \beta_2$ &0.9, 0.95 \\
Sequence length &4096 \\
Weight decay &0.1 \\
Scheduler &cosine decayed to 10\% \\
Warmup ratio &10\% \\
FSDP Strategy &\texttt{SHARD\_GRAD\_OP} \\
GPUs &64 \\
Precision &\texttt{bfloat16} \\
Batch size &1024 \\
Batch size (tokens) &4194304 \\
\bottomrule
\end{tabular}

D. Evaluation details

We provide additional evaluation dataset details in Table 11. The evaluation prompt setup is detailed in Figure 10. For discriminative benchmarks, we report accuracy and 95% confidence interval calculated from the standard error reported by lm-eval-harness. For FLORES-200 (MT) evaluation we calculate the confidence intervals using bootstrap resampling.

**Figure 10:** Prompts used for evaluation with `lm-eval-harness` [43].

E. Fast Vocabulary Transfer (FVT)

To formally define Fast Vocabulary Transfer [FVT 27], let $V$ and $tok$ denote the vocabulary and tokenization function of the original tokenizer, and let $E(t)$ denote the embedding of token $t$. For a new tokenizer with vocabulary $V'$, the embedding $E'(t)$ for each $t \in V'$ is defined as:

$ E'(t) = \begin{dcases} E(t) & \text{if } t \in V, \ \frac{\sum_{t_i \in tok(t)} E(t_i)}{|tok(t)|} & \text{otherwise}. \end{dcases}\tag{1} $

: Table 11: Evaluation datasets. n-shot is the number of in-context few-shot examples used for prompting during evaluation.

Dataset		Size	Split	n-shots	License
Winogrande	[38]	1267	valid	0	CC-BY
Winogrande ET	[39]	1767	test	0	Apache 2.0
XCOPA	[40]	600	test	5	CC-BY-4.0
SIB200	[41]	204	test	5	CC-BY-SA-4.0
FLORES200	[37]	1012	devtest	5	CC-BY-SA-4.0
Belebele	[42]	900	test	5	CC-BY-SA-4.0

F. Languages in training data

We provide the overview of the languages in the training dataset in Table 12.

\begin{tabular}{lllrllllr}\toprule
\textbf{Language} &\textbf{Code} &\textbf{Family} &\textbf{Bytes} &\textbf{} &\textbf{Language} &\textbf{Code} &\textbf{Family} &\textbf{Bytes} \\\cmidrule{1-4}\cmidrule{6-9}
Russian &rus\_Cyrl &Indo-European &6.4T & &Standard Estonian &ekk\_Latn &Uralic &43.8B \\
Mandarin Chinese &cmn\_Hani &Sino-Tibetan &2.7T & &Croatian &hrv\_Latn &Indo-European &38.6B \\
German &deu\_Latn &Indo-European &1.7T & &Standard Latvian &lvs\_Latn &Indo-European &35.8B \\
Japanese &jpn\_Jpan &Japonic &1.7T & &Standard Malay &zsm\_Latn &Austronesian &34.3B \\
Spanish &spa\_Latn &Indo-European &1.4T & &North Azerbaijani &azj\_Latn &Turkic &28.9B \\
French &fra\_Latn &Indo-European &1.2T & &Tamil &tam\_Taml &Dravidian &39.7B \\
Italian &ita\_Latn &Indo-European &793.8B & &Serbian &srp\_Cyrl &Indo-European &28.8B \\
Portuguese &por\_Latn &Indo-European &611.2B & &Tosk Albanian &als\_Latn &Indo-European &19.5B \\
Polish &pol\_Latn &Indo-European &463.9B & &Urdu &urd\_Arab &Indo-European &21.4B \\
Dutch &nld\_Latn &Indo-European &426.8B & &Kazakh &kaz\_Cyrl &Turkic &22.2B \\
Indonesian &ind\_Latn &Austronesian &374.4B & &Georgian &kat\_Geor &Kartvelian &27.1B \\
Turkish &tur\_Latn &Turkic &305.5B & &Nepali (individual language) &npi\_Deva &Indo-European &27.0B \\
Vietnamese &vie\_Latn &Austro-Asiatic &343.4B & &Marathi &mar\_Deva &Indo-European &24.2B \\
Czech &ces\_Latn &Indo-European &221.5B & &Malayalam &mal\_Mlym &Dravidian &23.9B \\
Standard Arabic &arb\_Arab &Afro-Asiatic &315.2B & &Macedonian &mkd\_Cyrl &Indo-European &16.1B \\
Korean &kor\_Hang &Koreanic &229.2B & &Icelandic &isl\_Latn &Indo-European &11.0B \\
Persian &fas\_Arab &Indo-European &327.1B & &Belarusian &bel\_Cyrl &Indo-European &12.3B \\
Hungarian &hun\_Latn &Uralic &214.4B & &Telugu &tel\_Telu &Dravidian &15.5B \\
Swedish &swe\_Latn &Indo-European &217.9B & &Afrikaans &afr\_Latn &Indo-European &8.3B \\
Romanian &ron\_Latn &Indo-European &199.9B & &Kannada &kan\_Knda &Dravidian &13.9B \\
Ukrainian &ukr\_Cyrl &Indo-European &273.7B & &Gujarati &guj\_Gujr &Indo-European &12.6B \\
Norwegian Bokmål &nob\_Latn &Indo-European &184.7B & &Galician &glg\_Latn &Indo-European &6.9B \\
Modern Greek (1453-) &ell\_Grek &Indo-European &238.4B & &Burmese &mya\_Mymr &Sino-Tibetan &13.3B \\
Thai &tha\_Thai &Kra-Dai &299.2B & &Moroccan Arabic &ary\_Arab &Afro-Asiatic &8.3B \\
Danish &dan\_Latn &Indo-European &161.8B & &Halh Mongolian &khk\_Cyrl &Mongolic &9.1B \\
Finnish &fin\_Latn &Uralic &153.6B & &Armenian &hye\_Armn &Indo-European &7.7B \\
Bulgarian &bul\_Cyrl &Indo-European &156.5B & &Khmer &khm\_Khmr &Austro-Asiatic &9.3B \\
Slovak &slk\_Latn &Indo-European &91.7B & &Northern Uzbek &uzn\_Latn &Turkic &4.8B \\
Hindi &hin\_Deva &Indo-European &129.9B & &Basque &eus\_Latn &Language isolate &4.6B \\
Lithuanian &lit\_Latn &Indo-European &60.7B & &Sinhala &sin\_Sinh &Indo-European &7.6B \\
Bosnian &bos\_Latn &Indo-European &52.8B & &Panjabi &pan\_Guru &Indo-European &6.1B \\
Hebrew &heb\_Hebr &Afro-Asiatic &73.8B & &Kirghiz &kir\_Cyrl &Turkic &4.7B \\
Bengali &ben\_Beng &Indo-European &93.5B & &Swahili (individual language) &swh\_Latn &Niger-Congo &3.3B \\
Slovenian &slv\_Latn &Indo-European &44.9B & &Norwegian Nynorsk &nno\_Latn &Indo-European &2.9B \\
Catalan &cat\_Latn &Indo-European &43.3B & &Odia &ory\_Orya &Indo-European &5.3B \\
\bottomrule
\end{tabular}

G. Full results

G.1 Extended tokenizer pruning results

We provide pruning compression and unreachable token curves for additional languages in Figure 12 and Figure 13, respectively. Additionally, we provide downstream evaluation of pruning for German in Figure 11, where our pruning methods perform the best.

**Figure 11:** Llama-3.1-8B vocabulary pruned with different methods and evaluated on German and English downstream tasks. The dataset for training the pruner consisted of 50-50 German-English mix. The highlighted area shows the 95% confidence interval. Note that *Freq (leaf)* and *Merge-based* method results overlap.

$**Figure 12:** Change in **compression** (*bytes per token* $\uparrow$) depending on the pruning method. Note that *Freq (leaf)* and *Merge-based* method results overlap.$

$**Figure 13:** Number of **unreachable tokens $\downarrow$** created by different pruning methods.$

G.2 Full tokenizer extension results

We provide the full improvement of our method (continued BPE training) over the naive method in Table 13 and Table 14 for compression and in Table 15 and Table 16 for unreachable tokens. We list full compression values for continued BPE training in Table 17 and Table 18.

\begin{tabular}{lcccccccccccccc}\toprule
&\multicolumn{6}{c}{\textbf{Llama-3 extension}} & &\multicolumn{6}{c}{\textbf{Qwen-2.5 extension}} \\\cmidrule{2-7}\cmidrule{9-14}
\textbf{Lang.} &\textbf{1000} &\textbf{2000} &\textbf{4000} &\textbf{8000} &\textbf{16000} &\textbf{32000} & &\textbf{1000} &\textbf{2000} &\textbf{4000} &\textbf{8000} &\textbf{16000} &\textbf{32000} \\\midrule
afr\_Latn &\cellcolor[HTML]{f9f8fe}1.9 &\cellcolor[HTML]{f8f8fd}2.1 &\cellcolor[HTML]{f7f7fd}2.3 &\cellcolor[HTML]{f9f9fe}1.7 &\cellcolor[HTML]{fbfafe}1.4 &\cellcolor[HTML]{fbfbfe}1.2 & &\cellcolor[HTML]{f3f2fc}3.4 &\cellcolor[HTML]{f0effb}4.4 &\cellcolor[HTML]{ebeafa}5.6 &\cellcolor[HTML]{e8e6f9}6.6 &\cellcolor[HTML]{e6e4f8}7.2 &\cellcolor[HTML]{e6e4f8}7.2 \\
als\_Latn &\cellcolor[HTML]{eeedfb}5.0 &\cellcolor[HTML]{efeefb}4.6 &\cellcolor[HTML]{f1f0fb}4.1 &\cellcolor[HTML]{f2f1fc}3.9 &\cellcolor[HTML]{f6f6fd}2.6 &\cellcolor[HTML]{f9f8fe}1.9 & &\cellcolor[HTML]{e3e1f8}8.0 &\cellcolor[HTML]{dedbf6}9.5 &\cellcolor[HTML]{d9d6f5}10.8 &\cellcolor[HTML]{d9d6f5}11.0 &\cellcolor[HTML]{dcdaf6}10.0 &\cellcolor[HTML]{dcd9f6}10.1 \\
arb\_Arab &\cellcolor[HTML]{fcfcff}0.9 &\cellcolor[HTML]{fcfcff}1.0 &\cellcolor[HTML]{fcfbfe}1.1 &\cellcolor[HTML]{fbfbfe}1.3 &\cellcolor[HTML]{fcfcff}0.9 &\cellcolor[HTML]{fdfdff}0.6 & &\cellcolor[HTML]{faf9fe}1.5 &\cellcolor[HTML]{f8f8fd}2.0 &\cellcolor[HTML]{f5f4fd}2.8 &\cellcolor[HTML]{f4f3fc}3.2 &\cellcolor[HTML]{f3f2fc}3.5 &\cellcolor[HTML]{f2f1fc}3.8 \\
ary\_Arab &\cellcolor[HTML]{fefeff}0.4 &\cellcolor[HTML]{fcfcff}0.9 &\cellcolor[HTML]{fcfcff}0.9 &\cellcolor[HTML]{fcfcff}0.9 &\cellcolor[HTML]{fcfcff}0.9 &\cellcolor[HTML]{fdfdff}0.6 & &\cellcolor[HTML]{fdfdff}0.6 &\cellcolor[HTML]{fafafe}1.3 &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{f9f9fe}1.7 &\cellcolor[HTML]{f7f6fd}2.3 &\cellcolor[HTML]{f7f6fd}2.4 \\
azj\_Latn &\cellcolor[HTML]{e2dff7}8.5 &\cellcolor[HTML]{dddaf6}9.9 &\cellcolor[HTML]{dad7f5}10.6 &\cellcolor[HTML]{dddaf6}9.8 &\cellcolor[HTML]{dfdcf6}9.3 &\cellcolor[HTML]{e6e4f8}7.4 & &\cellcolor[HTML]{d8d5f5}11.1 &\cellcolor[HTML]{cfccf2}13.6 &\cellcolor[HTML]{c1bdee}17.7 &\cellcolor[HTML]{bbb6ec}19.5 &\cellcolor[HTML]{afaae9}22.7 &\cellcolor[HTML]{aba5e8}23.9 \\
bel\_Cyrl &\cellcolor[HTML]{f3f2fc}3.7 &\cellcolor[HTML]{f0effb}4.5 &\cellcolor[HTML]{eeedfb}4.8 &\cellcolor[HTML]{efeefb}4.6 &\cellcolor[HTML]{f3f2fc}3.6 &\cellcolor[HTML]{f6f6fd}2.6 & &\cellcolor[HTML]{f2f1fc}3.8 &\cellcolor[HTML]{edecfa}5.2 &\cellcolor[HTML]{e8e6f9}6.7 &\cellcolor[HTML]{e2e0f7}8.4 &\cellcolor[HTML]{dfddf7}9.0 &\cellcolor[HTML]{dedcf6}9.4 \\
ben\_Beng &\cellcolor[HTML]{fefefe}-0.0 &0.0 &0.0 &0.0 &0.0 &0.0 & &0.0 &0.0 &0.0 &0.0 &0.0 &0.0 \\
bos\_Latn &\cellcolor[HTML]{e5e3f8}7.5 &\cellcolor[HTML]{dedcf6}9.5 &\cellcolor[HTML]{dcdaf6}9.9 &\cellcolor[HTML]{dcd9f6}10.2 &\cellcolor[HTML]{dfddf7}9.2 &\cellcolor[HTML]{e3e1f8}8.1 & &\cellcolor[HTML]{dfddf7}9.1 &\cellcolor[HTML]{d6d3f4}11.8 &\cellcolor[HTML]{c9c5f0}15.4 &\cellcolor[HTML]{bcb7ed}19.1 &\cellcolor[HTML]{afa9e9}22.9 &\cellcolor[HTML]{a19ae5}26.8 \\
bul\_Cyrl &\cellcolor[HTML]{e4e2f8}7.8 &\cellcolor[HTML]{e1def7}8.8 &\cellcolor[HTML]{dddbf6}9.7 &\cellcolor[HTML]{dcd9f6}10.1 &\cellcolor[HTML]{e0def7}9.0 &\cellcolor[HTML]{e6e4f8}7.4 & &\cellcolor[HTML]{d2cef3}13.0 &\cellcolor[HTML]{c6c2f0}16.3 &\cellcolor[HTML]{b7b2ec}20.5 &\cellcolor[HTML]{aaa4e8}24.2 &\cellcolor[HTML]{9e97e5}27.6 &\cellcolor[HTML]{9992e3}29.1 \\
cat\_Latn &\cellcolor[HTML]{f6f5fd}2.7 &\cellcolor[HTML]{f6f5fd}2.8 &\cellcolor[HTML]{f7f6fd}2.5 &\cellcolor[HTML]{f8f7fd}2.2 &\cellcolor[HTML]{f9f9fe}1.7 &\cellcolor[HTML]{fcfcfe}1.0 & &\cellcolor[HTML]{f3f2fc}3.6 &\cellcolor[HTML]{f0effb}4.3 &\cellcolor[HTML]{eeedfb}4.9 &\cellcolor[HTML]{edecfa}5.2 &\cellcolor[HTML]{ebe9fa}5.7 &\cellcolor[HTML]{ebeafa}5.6 \\
ces\_Latn &\cellcolor[HTML]{f4f3fc}3.4 &\cellcolor[HTML]{f1effb}4.3 &\cellcolor[HTML]{f0effb}4.3 &\cellcolor[HTML]{edecfa}5.3 &\cellcolor[HTML]{ecebfa}5.5 &\cellcolor[HTML]{eeedfb}5.0 & &\cellcolor[HTML]{ebe9fa}5.8 &\cellcolor[HTML]{e2e0f7}8.2 &\cellcolor[HTML]{dcd9f6}10.0 &\cellcolor[HTML]{d5d2f4}12.0 &\cellcolor[HTML]{cdcaf2}14.2 &\cellcolor[HTML]{c8c5f0}15.6 \\
cmn\_Hani &\cellcolor[HTML]{fefefe}-0.0 &\cellcolor[HTML]{fefefe}-0.0 &0.1 &0.0 &\cellcolor[HTML]{fefefe}-0.0 &\cellcolor[HTML]{fefefe}-0.1 & &0.0 &\cellcolor[HTML]{fffeff}0.2 &\cellcolor[HTML]{fefeff}0.2 &\cellcolor[HTML]{fefeff}0.3 &\cellcolor[HTML]{fefeff}0.2 &\cellcolor[HTML]{fefeff}0.3 \\
dan\_Latn &\cellcolor[HTML]{f8f8fd}2.1 &\cellcolor[HTML]{f7f7fd}2.3 &\cellcolor[HTML]{f7f7fd}2.3 &\cellcolor[HTML]{f7f7fd}2.3 &\cellcolor[HTML]{f8f8fe}2.1 &\cellcolor[HTML]{f9f9fe}1.8 & &\cellcolor[HTML]{f5f5fd}2.8 &\cellcolor[HTML]{f3f2fc}3.6 &\cellcolor[HTML]{f0eefb}4.4 &\cellcolor[HTML]{edecfa}5.2 &\cellcolor[HTML]{ecebfa}5.3 &\cellcolor[HTML]{ebe9fa}5.8 \\
deu\_Latn &\cellcolor[HTML]{f8f7fd}2.2 &\cellcolor[HTML]{f6f5fd}2.8 &\cellcolor[HTML]{f5f5fd}2.9 &\cellcolor[HTML]{f6f5fd}2.7 &\cellcolor[HTML]{f6f6fd}2.6 &\cellcolor[HTML]{f7f7fd}2.3 & &\cellcolor[HTML]{f7f6fd}2.3 &\cellcolor[HTML]{f4f3fc}3.1 &\cellcolor[HTML]{f1f0fb}4.1 &\cellcolor[HTML]{edecfb}5.0 &\cellcolor[HTML]{eae8f9}6.1 &\cellcolor[HTML]{e7e5f9}6.9 \\
ekk\_Latn &\cellcolor[HTML]{f1f0fb}4.1 &\cellcolor[HTML]{efedfb}4.8 &\cellcolor[HTML]{ecebfa}5.5 &\cellcolor[HTML]{eae9fa}6.0 &\cellcolor[HTML]{ebe9fa}5.9 &\cellcolor[HTML]{ebeafa}5.7 & &\cellcolor[HTML]{ecebfa}5.4 &\cellcolor[HTML]{e7e6f9}6.7 &\cellcolor[HTML]{e2e0f7}8.2 &\cellcolor[HTML]{dddbf6}9.6 &\cellcolor[HTML]{d8d5f5}11.1 &\cellcolor[HTML]{d3d0f3}12.5 \\
ell\_Grek &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{fafafe}1.6 &\cellcolor[HTML]{f9f8fe}1.9 &\cellcolor[HTML]{f9f8fe}1.9 &\cellcolor[HTML]{fbfbfe}1.3 &\cellcolor[HTML]{fdfdff}0.7 & &0.1 &\cellcolor[HTML]{fffeff}0.2 &\cellcolor[HTML]{fefeff}0.3 &\cellcolor[HTML]{fefeff}0.3 &\cellcolor[HTML]{fefdff}0.4 &0.0 \\
eus\_Latn &\cellcolor[HTML]{eae8fa}6.1 &\cellcolor[HTML]{e6e4f9}7.2 &\cellcolor[HTML]{e3e1f8}8.0 &\cellcolor[HTML]{e3e1f8}8.1 &\cellcolor[HTML]{e6e5f9}7.1 &\cellcolor[HTML]{eae8f9}6.2 & &\cellcolor[HTML]{dbd8f5}10.4 &\cellcolor[HTML]{cfcbf2}13.7 &\cellcolor[HTML]{c2beee}17.4 &\cellcolor[HTML]{b8b3ec}20.1 &\cellcolor[HTML]{b4afeb}21.2 &\cellcolor[HTML]{aca6e8}23.6 \\
fas\_Arab &\cellcolor[HTML]{fdfdff}0.7 &\cellcolor[HTML]{fdfdff}0.6 &\cellcolor[HTML]{fefdff}0.5 &0.2 &0.2 &0.1 & &\cellcolor[HTML]{f4f3fc}3.2 &\cellcolor[HTML]{f0effb}4.4 &\cellcolor[HTML]{ecebfa}5.3 &\cellcolor[HTML]{e9e7f9}6.4 &\cellcolor[HTML]{ecebfa}5.4 &\cellcolor[HTML]{f0effb}4.2 \\
fin\_Latn &\cellcolor[HTML]{e7e5f9}6.9 &\cellcolor[HTML]{e1dff7}8.6 &\cellcolor[HTML]{dddbf6}9.7 &\cellcolor[HTML]{dad7f5}10.6 &\cellcolor[HTML]{d8d5f5}11.2 &\cellcolor[HTML]{d9d6f5}10.9 & &\cellcolor[HTML]{e4e2f8}7.7 &\cellcolor[HTML]{dad7f5}10.6 &\cellcolor[HTML]{d1cef3}13.0 &\cellcolor[HTML]{c7c2f0}16.1 &\cellcolor[HTML]{bbb6ed}19.3 &\cellcolor[HTML]{b3adea}21.7 \\
fra\_Latn &\cellcolor[HTML]{f8f8fd}2.1 &\cellcolor[HTML]{f8f8fd}2.1 &\cellcolor[HTML]{f8f8fe}2.0 &\cellcolor[HTML]{faf9fe}1.7 &\cellcolor[HTML]{fbfafe}1.4 &\cellcolor[HTML]{fdfcff}0.8 & &\cellcolor[HTML]{f7f7fd}2.3 &\cellcolor[HTML]{f6f6fd}2.5 &\cellcolor[HTML]{f5f4fd}2.8 &\cellcolor[HTML]{f3f2fc}3.5 &\cellcolor[HTML]{f0effb}4.2 &\cellcolor[HTML]{f1f0fb}4.0 \\
glg\_Latn &\cellcolor[HTML]{f9f9fe}1.9 &\cellcolor[HTML]{f9f8fe}2.0 &\cellcolor[HTML]{f8f7fd}2.2 &\cellcolor[HTML]{f9f9fe}1.8 &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{fcfcfe}1.0 & &\cellcolor[HTML]{f7f6fd}2.4 &\cellcolor[HTML]{f5f4fd}2.8 &\cellcolor[HTML]{f3f2fc}3.5 &\cellcolor[HTML]{f2f1fc}3.7 &\cellcolor[HTML]{f0effb}4.3 &\cellcolor[HTML]{efeefb}4.4 \\
guj\_Gujr &0.0 &0.0 &0.0 &0.0 &0.0 &\cellcolor[HTML]{fefefe}-0.0 & &0.0 &0.0 &0.0 &0.0 &0.0 &0.0 \\
heb\_Hebr &\cellcolor[HTML]{9890e3}29.5 &\cellcolor[HTML]{a099e5}27.1 &\cellcolor[HTML]{aca6e8}23.7 &\cellcolor[HTML]{bab6ec}19.6 &\cellcolor[HTML]{ccc8f1}14.7 &\cellcolor[HTML]{dfddf7}9.1 & &\cellcolor[HTML]{dddaf6}9.8 &\cellcolor[HTML]{cecaf2}14.1 &\cellcolor[HTML]{c0bcee}17.9 &\cellcolor[HTML]{b4afeb}21.3 &\cellcolor[HTML]{a8a2e7}24.7 &\cellcolor[HTML]{9f98e5}27.3 \\
hin\_Deva &0.0 &0.0 &0.0 &0.0 &0.0 &0.0 & &0.0 &0.0 &0.0 &0.0 &0.0 &0.0 \\
hrv\_Latn &\cellcolor[HTML]{e2e0f7}8.3 &\cellcolor[HTML]{dedcf6}9.4 &\cellcolor[HTML]{d9d6f5}11.1 &\cellcolor[HTML]{d7d4f4}11.4 &\cellcolor[HTML]{dad8f5}10.6 &\cellcolor[HTML]{e2dff7}8.5 & &\cellcolor[HTML]{dddbf6}9.7 &\cellcolor[HTML]{d1cef3}13.1 &\cellcolor[HTML]{c2beef}17.4 &\cellcolor[HTML]{b2adea}21.8 &\cellcolor[HTML]{a39ce6}26.2 &\cellcolor[HTML]{958de2}30.3 \\
hun\_Latn &\cellcolor[HTML]{f1f0fb}4.1 &\cellcolor[HTML]{edecfa}5.2 &\cellcolor[HTML]{e9e8f9}6.2 &\cellcolor[HTML]{e7e5f9}7.0 &\cellcolor[HTML]{e7e5f9}7.1 &\cellcolor[HTML]{e8e6f9}6.7 & &\cellcolor[HTML]{edecfa}5.1 &\cellcolor[HTML]{e6e4f8}7.2 &\cellcolor[HTML]{dfdcf6}9.3 &\cellcolor[HTML]{d4d1f3}12.3 &\cellcolor[HTML]{cac7f1}15.0 &\cellcolor[HTML]{c3bfef}17.0 \\
hye\_Armn &0.0 &0.0 &\cellcolor[HTML]{fefefe}-0.0 &0.0 &0.0 &0.1 & &\cellcolor[HTML]{fafafe}1.3 &\cellcolor[HTML]{fdfcff}0.7 &\cellcolor[HTML]{fdfcff}0.7 &\cellcolor[HTML]{fcfcff}0.8 &\cellcolor[HTML]{fcfcfe}0.9 &\cellcolor[HTML]{fdfdff}0.5 \\
ind\_Latn &\cellcolor[HTML]{f7f7fd}2.3 &\cellcolor[HTML]{f8f8fd}2.1 &\cellcolor[HTML]{f8f7fd}2.3 &\cellcolor[HTML]{fafafe}1.6 &\cellcolor[HTML]{fcfcff}0.9 &\cellcolor[HTML]{fdfdff}0.6 & &\cellcolor[HTML]{efedfb}4.7 &\cellcolor[HTML]{e9e8f9}6.2 &\cellcolor[HTML]{e2e0f7}8.2 &\cellcolor[HTML]{e0ddf7}9.0 &\cellcolor[HTML]{dedcf6}9.3 &\cellcolor[HTML]{e1dff7}8.6 \\
isl\_Latn &\cellcolor[HTML]{f7f6fd}2.5 &\cellcolor[HTML]{f5f4fd}3.0 &\cellcolor[HTML]{f3f2fc}3.5 &\cellcolor[HTML]{f3f2fc}3.5 &\cellcolor[HTML]{f1f0fb}4.2 &\cellcolor[HTML]{f1f0fc}4.0 & &\cellcolor[HTML]{f5f4fd}2.9 &\cellcolor[HTML]{f0effb}4.2 &\cellcolor[HTML]{ecebfa}5.5 &\cellcolor[HTML]{e9e7f9}6.3 &\cellcolor[HTML]{e2e0f7}8.2 &\cellcolor[HTML]{e0def7}8.8 \\
ita\_Latn &\cellcolor[HTML]{edebfa}5.4 &\cellcolor[HTML]{ecebfa}5.5 &\cellcolor[HTML]{ebeafa}5.7 &\cellcolor[HTML]{efeefb}4.7 &\cellcolor[HTML]{f4f3fc}3.3 &\cellcolor[HTML]{f7f6fd}2.5 & &\cellcolor[HTML]{eae9fa}5.9 &\cellcolor[HTML]{e6e4f8}7.3 &\cellcolor[HTML]{dfddf7}9.0 &\cellcolor[HTML]{dad7f5}10.7 &\cellcolor[HTML]{d4d1f3}12.3 &\cellcolor[HTML]{cfccf2}13.7 \\
jpn\_Jpan &\cellcolor[HTML]{fbfbfe}1.3 &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{fafafe}1.4 &\cellcolor[HTML]{faf9fe}1.7 &\cellcolor[HTML]{fbfbfe}1.2 & &\cellcolor[HTML]{fcfcfe}0.9 &\cellcolor[HTML]{fcfcff}0.9 &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fbfbfe}1.1 &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fbfbfe}1.0 \\
kan\_Knda &\cellcolor[HTML]{fefefe}-0.0 &\cellcolor[HTML]{fefefe}-0.0 &0.0 &0.0 &0.0 &0.0 & &0.0 &0.0 &0.0 &-0.1 &-0.1 &0.0 \\
kat\_Geor &0.2 &\cellcolor[HTML]{fefeff}0.4 &0.2 &0.2 &\cellcolor[HTML]{fffeff}0.3 &0.2 & &\cellcolor[HTML]{f9f8fe}1.9 &\cellcolor[HTML]{f7f6fd}2.4 &\cellcolor[HTML]{f7f6fd}2.3 &\cellcolor[HTML]{f5f5fd}2.8 &\cellcolor[HTML]{f3f2fc}3.5 &\cellcolor[HTML]{f3f2fc}3.4 \\
kaz\_Cyrl &\cellcolor[HTML]{ecebfa}5.4 &\cellcolor[HTML]{e8e6f9}6.7 &\cellcolor[HTML]{e8e6f9}6.7 &\cellcolor[HTML]{eae9fa}6.1 &\cellcolor[HTML]{eeedfb}5.0 &\cellcolor[HTML]{f3f2fc}3.5 & &\cellcolor[HTML]{e4e3f8}7.6 &\cellcolor[HTML]{dad8f5}10.5 &\cellcolor[HTML]{d3d0f3}12.4 &\cellcolor[HTML]{ccc9f1}14.4 &\cellcolor[HTML]{c9c5f0}15.5 &\cellcolor[HTML]{c6c2f0}16.2 \\
khk\_Cyrl &\cellcolor[HTML]{f2f1fc}3.8 &\cellcolor[HTML]{ecebfa}5.5 &\cellcolor[HTML]{edecfb}5.1 &\cellcolor[HTML]{f2f1fc}3.8 &\cellcolor[HTML]{f5f4fd}3.0 &\cellcolor[HTML]{f7f7fd}2.3 & &\cellcolor[HTML]{e6e4f8}7.1 &\cellcolor[HTML]{e0def7}8.9 &\cellcolor[HTML]{d6d3f4}11.8 &\cellcolor[HTML]{d5d2f4}12.0 &\cellcolor[HTML]{d7d4f4}11.5 &\cellcolor[HTML]{dbd8f5}10.4 \\
khm\_Khmr &0.0 &0.0 &0.0 &0.0 &0.0 &0.0 & &0.0 &0.0 &0.0 &0.0 &0.0 &0.0 \\
kir\_Cyrl &\cellcolor[HTML]{eae9fa}6.0 &\cellcolor[HTML]{e8e7f9}6.6 &\cellcolor[HTML]{e5e3f8}7.4 &\cellcolor[HTML]{e9e7f9}6.4 &\cellcolor[HTML]{ecebfa}5.4 &\cellcolor[HTML]{f1f0fb}4.1 & &\cellcolor[HTML]{e8e7f9}6.5 &\cellcolor[HTML]{e2e0f7}8.3 &\cellcolor[HTML]{d5d2f4}12.1 &\cellcolor[HTML]{d0cdf2}13.3 &\cellcolor[HTML]{cfccf2}13.6 &\cellcolor[HTML]{cecaf2}14.1 \\
kor\_Hang &\cellcolor[HTML]{fefeff}0.4 &\cellcolor[HTML]{fefeff}0.5 &\cellcolor[HTML]{fdfdff}0.6 &\cellcolor[HTML]{fdfdff}0.7 &\cellcolor[HTML]{fdfdff}0.8 &\cellcolor[HTML]{fdfdff}0.8 & &\cellcolor[HTML]{fdfdff}0.6 &\cellcolor[HTML]{fcfcff}0.8 &\cellcolor[HTML]{fcfcff}0.9 &\cellcolor[HTML]{fcfbfe}1.0 &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fafafe}1.3 \\
lit\_Latn &\cellcolor[HTML]{f0effb}4.4 &\cellcolor[HTML]{edecfa}5.1 &\cellcolor[HTML]{eceafa}5.6 &\cellcolor[HTML]{ebe9fa}5.9 &\cellcolor[HTML]{eceafa}5.6 &\cellcolor[HTML]{efeefb}4.7 & &\cellcolor[HTML]{eeedfb}4.8 &\cellcolor[HTML]{e7e5f9}6.9 &\cellcolor[HTML]{e0ddf7}9.0 &\cellcolor[HTML]{d5d2f4}11.9 &\cellcolor[HTML]{cdc9f1}14.4 &\cellcolor[HTML]{c7c3f0}16.0 \\
lvs\_Latn &\cellcolor[HTML]{f2f1fc}4.0 &\cellcolor[HTML]{eeedfb}5.0 &\cellcolor[HTML]{eeedfb}4.8 &\cellcolor[HTML]{eeecfb}5.1 &\cellcolor[HTML]{efeefb}4.6 &\cellcolor[HTML]{f3f2fc}3.7 & &\cellcolor[HTML]{edebfa}5.2 &\cellcolor[HTML]{e5e3f8}7.4 &\cellcolor[HTML]{e2e0f7}8.3 &\cellcolor[HTML]{dddbf6}9.6 &\cellcolor[HTML]{d8d5f5}11.2 &\cellcolor[HTML]{d5d2f4}12.1 \\
mal\_Mlym &0.0 &0.0 &0.0 &0.0 &0.0 &0.0 & &0.0 &0.1 &0.1 &0.1 &0.0 &0.0 \\
mar\_Deva &0.0 &0.0 &0.0 &0.0 &0.0 &0.0 & &0.1 &0.0 &0.0 &0.0 &0.0 &0.0 \\
mkd\_Cyrl &\cellcolor[HTML]{e3e1f8}8.1 &\cellcolor[HTML]{dddbf6}9.7 &\cellcolor[HTML]{dad7f5}10.7 &\cellcolor[HTML]{dcdaf6}10.0 &\cellcolor[HTML]{e2e0f7}8.5 &\cellcolor[HTML]{e9e7f9}6.4 & &\cellcolor[HTML]{d4d1f3}12.4 &\cellcolor[HTML]{c5c1ef}16.6 &\cellcolor[HTML]{b6b0eb}20.9 &\cellcolor[HTML]{a9a3e8}24.4 &\cellcolor[HTML]{9c95e4}28.3 &\cellcolor[HTML]{9891e3}29.3 \\
mya\_Mymr &\cellcolor[HTML]{fefefe}-0.1 &0.0 &0.0 &0.0 &0.0 &0.0 & &\cellcolor[HTML]{fdfdff}0.5 &\cellcolor[HTML]{fdfdff}0.6 &\cellcolor[HTML]{fdfdff}0.6 &\cellcolor[HTML]{fdfdff}0.6 &\cellcolor[HTML]{fdfdff}0.6 &\cellcolor[HTML]{fdfdff}0.6 \\
nld\_Latn &\cellcolor[HTML]{f8f8fe}2.0 &\cellcolor[HTML]{f8f7fd}2.2 &\cellcolor[HTML]{f8f7fd}2.2 &\cellcolor[HTML]{f8f7fd}2.2 &\cellcolor[HTML]{faf9fe}1.7 &\cellcolor[HTML]{fbfafe}1.3 & &\cellcolor[HTML]{f2f1fc}3.6 &\cellcolor[HTML]{f0effb}4.4 &\cellcolor[HTML]{eceafa}5.6 &\cellcolor[HTML]{e7e5f9}6.9 &\cellcolor[HTML]{e5e4f8}7.3 &\cellcolor[HTML]{e5e3f8}7.5 \\
nno\_Latn &\cellcolor[HTML]{f9f8fe}1.9 &\cellcolor[HTML]{f8f7fd}2.2 &\cellcolor[HTML]{f7f6fd}2.5 &\cellcolor[HTML]{f7f7fd}2.3 &\cellcolor[HTML]{f8f7fd}2.2 &\cellcolor[HTML]{f9f9fe}1.8 & &\cellcolor[HTML]{f3f2fc}3.6 &\cellcolor[HTML]{f0effb}4.4 &\cellcolor[HTML]{f1f0fb}4.1 &\cellcolor[HTML]{f0effb}4.4 &\cellcolor[HTML]{eeedfb}4.9 &\cellcolor[HTML]{ecebfa}5.4 \\
nob\_Latn &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{f9f9fe}1.7 &\cellcolor[HTML]{f8f8fe}2.0 &\cellcolor[HTML]{f9f8fe}1.9 &\cellcolor[HTML]{fafafe}1.6 &\cellcolor[HTML]{faf9fe}1.7 & &\cellcolor[HTML]{f7f7fd}2.2 &\cellcolor[HTML]{f5f5fd}2.7 &\cellcolor[HTML]{f2f1fc}3.8 &\cellcolor[HTML]{f0effb}4.2 &\cellcolor[HTML]{efedfb}4.7 &\cellcolor[HTML]{edebfa}5.3 \\
npi\_Deva &\cellcolor[HTML]{fefefe}-0.0 &0.0 &0.0 &0.0 &0.0 &0.0 & &0.0 &-0.0 &0.0 &0.0 &0.0 &0.0 \\
ory\_Orya &0.0 &0.0 &0.0 &0.0 &0.0 &0.0 & &\cellcolor[HTML]{fffeff}0.2 &\cellcolor[HTML]{fffeff}0.2 &\cellcolor[HTML]{fefeff}0.2 &0.0 &0.0 &-0.1 \\
pan\_Guru &0.0 &0.2 &0.2 &0.2 &0.2 &0.2 & &-0.0 &-0.0 &0.0 &0.0 &0.0 &0.0 \\
pol\_Latn &\cellcolor[HTML]{e9e7f9}6.5 &\cellcolor[HTML]{e0def7}9.0 &\cellcolor[HTML]{d9d6f5}11.0 &\cellcolor[HTML]{d7d4f4}11.6 &\cellcolor[HTML]{d9d7f5}10.8 &\cellcolor[HTML]{e0ddf7}9.0 & &\cellcolor[HTML]{e9e8f9}6.2 &\cellcolor[HTML]{dedcf6}9.4 &\cellcolor[HTML]{d2cff3}12.8 &\cellcolor[HTML]{c6c2f0}16.3 &\cellcolor[HTML]{b9b4ec}19.8 &\cellcolor[HTML]{aca7e9}23.5 \\
por\_Latn &\cellcolor[HTML]{fbfafe}1.4 &\cellcolor[HTML]{f9f9fe}1.8 &\cellcolor[HTML]{faf9fe}1.7 &\cellcolor[HTML]{fbfafe}1.4 &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fcfcff}0.9 & &\cellcolor[HTML]{f9f9fe}1.6 &\cellcolor[HTML]{f7f6fd}2.3 &\cellcolor[HTML]{f6f5fd}2.7 &\cellcolor[HTML]{f4f3fc}3.1 &\cellcolor[HTML]{f2f1fc}3.8 &\cellcolor[HTML]{f2f1fc}3.8 \\
ron\_Latn &\cellcolor[HTML]{ecebfa}5.5 &\cellcolor[HTML]{edecfa}5.2 &\cellcolor[HTML]{eceafa}5.6 &\cellcolor[HTML]{edebfa}5.3 &\cellcolor[HTML]{f0effb}4.4 &\cellcolor[HTML]{f4f4fc}3.1 & &\cellcolor[HTML]{e8e6f9}6.6 &\cellcolor[HTML]{e5e3f8}7.6 &\cellcolor[HTML]{e0ddf7}9.0 &\cellcolor[HTML]{dbd9f5}10.2 &\cellcolor[HTML]{d8d5f5}11.2 &\cellcolor[HTML]{d8d5f5}11.2 \\
rus\_Cyrl &\cellcolor[HTML]{f7f7fd}2.3 &\cellcolor[HTML]{f8f7fd}2.2 &\cellcolor[HTML]{f7f6fd}2.5 &\cellcolor[HTML]{f6f5fd}2.7 &\cellcolor[HTML]{f6f6fd}2.6 &\cellcolor[HTML]{f8f7fd}2.2 & &\cellcolor[HTML]{f8f8fd}2.0 &\cellcolor[HTML]{f7f6fd}2.4 &\cellcolor[HTML]{f5f4fd}2.9 &\cellcolor[HTML]{f2f2fc}3.6 &\cellcolor[HTML]{f1f0fb}4.1 &\cellcolor[HTML]{efeefb}4.6 \\
sin\_Sinh &\cellcolor[HTML]{fcfcff}0.9 &\cellcolor[HTML]{fefeff}0.5 &0.2 &0.1 &0.1 &0.0 & &\cellcolor[HTML]{bfbaee}18.4 &\cellcolor[HTML]{bbb6ed}19.3 &\cellcolor[HTML]{b9b4ec}19.9 &\cellcolor[HTML]{b9b4ec}20.0 &\cellcolor[HTML]{b8b3ec}20.2 &\cellcolor[HTML]{b9b4ec}20.0 \\
slk\_Latn &\cellcolor[HTML]{ebe9fa}5.9 &\cellcolor[HTML]{e7e5f9}7.0 &\cellcolor[HTML]{e6e5f9}7.1 &\cellcolor[HTML]{e5e3f8}7.6 &\cellcolor[HTML]{e6e4f8}7.2 &\cellcolor[HTML]{e9e7f9}6.5 & &\cellcolor[HTML]{e9e7f9}6.3 &\cellcolor[HTML]{dedcf6}9.3 &\cellcolor[HTML]{d5d2f4}12.1 &\cellcolor[HTML]{cecaf2}14.1 &\cellcolor[HTML]{c3beef}17.2 &\cellcolor[HTML]{beb9ed}18.6 \\
slv\_Latn &\cellcolor[HTML]{e7e5f9}7.0 &\cellcolor[HTML]{e3e1f8}8.2 &\cellcolor[HTML]{dfddf7}9.1 &\cellcolor[HTML]{dedcf6}9.4 &\cellcolor[HTML]{e1dff7}8.6 &\cellcolor[HTML]{e6e4f9}7.2 & &\cellcolor[HTML]{e1dff7}8.6 &\cellcolor[HTML]{d9d6f5}10.8 &\cellcolor[HTML]{cfcbf2}13.7 &\cellcolor[HTML]{c4c0ef}16.8 &\cellcolor[HTML]{bab5ec}19.6 &\cellcolor[HTML]{b2acea}22.0 \\
spa\_Latn &\cellcolor[HTML]{f5f5fd}2.9 &\cellcolor[HTML]{f5f5fd}2.9 &\cellcolor[HTML]{f7f6fd}2.5 &\cellcolor[HTML]{f8f7fd}2.2 &\cellcolor[HTML]{faf9fe}1.6 &\cellcolor[HTML]{fcfbfe}1.1 & &\cellcolor[HTML]{f4f4fc}3.0 &\cellcolor[HTML]{f4f3fc}3.2 &\cellcolor[HTML]{f3f3fc}3.3 &\cellcolor[HTML]{f1f0fc}4.0 &\cellcolor[HTML]{efeefb}4.5 &\cellcolor[HTML]{efeefb}4.7 \\
srp\_Cyrl &\cellcolor[HTML]{f5f4fd}2.9 &\cellcolor[HTML]{f3f2fc}3.6 &\cellcolor[HTML]{f0effb}4.4 &\cellcolor[HTML]{f1f0fc}4.0 &\cellcolor[HTML]{f3f2fc}3.6 &\cellcolor[HTML]{f6f5fd}2.8 & &\cellcolor[HTML]{f4f3fc}3.1 &\cellcolor[HTML]{eeedfb}4.8 &\cellcolor[HTML]{eae8f9}6.1 &\cellcolor[HTML]{e5e3f8}7.6 &\cellcolor[HTML]{dfddf7}9.1 &\cellcolor[HTML]{dddaf6}9.8 \\
swe\_Latn &\cellcolor[HTML]{faf9fe}1.6 &\cellcolor[HTML]{f9f8fe}1.9 &\cellcolor[HTML]{f8f8fd}2.1 &\cellcolor[HTML]{f7f7fd}2.3 &\cellcolor[HTML]{f8f8fd}2.1 &\cellcolor[HTML]{f9f8fe}1.9 & &\cellcolor[HTML]{f7f6fd}2.4 &\cellcolor[HTML]{f4f4fc}3.0 &\cellcolor[HTML]{f2f1fc}3.8 &\cellcolor[HTML]{eeedfb}4.8 &\cellcolor[HTML]{eceafa}5.5 &\cellcolor[HTML]{eae9fa}5.9 \\
swh\_Latn &\cellcolor[HTML]{dfddf7}9.2 &\cellcolor[HTML]{d9d6f5}10.9 &\cellcolor[HTML]{d9d6f5}10.9 &\cellcolor[HTML]{d9d6f5}10.9 &\cellcolor[HTML]{dedcf6}9.4 &\cellcolor[HTML]{e5e3f8}7.5 & &\cellcolor[HTML]{c5c1ef}16.4 &\cellcolor[HTML]{aea8e9}23.2 &\cellcolor[HTML]{958de2}30.2 &\cellcolor[HTML]{8077dc}36.2 &\cellcolor[HTML]{7369d9}39.9 &\cellcolor[HTML]{7268d8}40.8 \\
tam\_Taml &0.0 &0.0 &0.0 &0.0 &0.0 &\cellcolor[HTML]{fefefe}-0.0 & &0.0 &0.0 &0.0 &0.0 &0.0 &-0.0 \\
tel\_Telu &0.0 &0.0 &0.0 &0.0 &0.0 &0.0 & &0.0 &0.1 &0.0 &0.0 &0.0 &0.0 \\
tha\_Thai &\cellcolor[HTML]{fefeff}0.4 &\cellcolor[HTML]{fefeff}0.4 &\cellcolor[HTML]{fdfdff}0.6 &\cellcolor[HTML]{fefeff}0.4 &\cellcolor[HTML]{fefeff}0.4 &\cellcolor[HTML]{fefeff}0.5 & &\cellcolor[HTML]{fbfbfe}1.1 &\cellcolor[HTML]{fcfcff}0.7 &\cellcolor[HTML]{fbfbfe}1.1 &\cellcolor[HTML]{fbfbfe}1.1 &\cellcolor[HTML]{fbfbfe}1.1 &\cellcolor[HTML]{fbfbfe}1.1 \\
tur\_Latn &\cellcolor[HTML]{eeedfb}5.0 &\cellcolor[HTML]{edecfa}5.3 &\cellcolor[HTML]{eceafa}5.6 &\cellcolor[HTML]{eae9fa}6.0 &\cellcolor[HTML]{ebeafa}5.7 &\cellcolor[HTML]{efeefb}4.6 & &\cellcolor[HTML]{f2f1fc}3.8 &\cellcolor[HTML]{eceafa}5.5 &\cellcolor[HTML]{e5e3f8}7.5 &\cellcolor[HTML]{dfddf7}9.1 &\cellcolor[HTML]{d9d7f5}10.7 &\cellcolor[HTML]{d6d3f4}11.8 \\
ukr\_Cyrl &\cellcolor[HTML]{f7f6fd}2.5 &\cellcolor[HTML]{f7f6fd}2.4 &\cellcolor[HTML]{f6f5fd}2.7 &\cellcolor[HTML]{f6f5fd}2.8 &\cellcolor[HTML]{f6f6fd}2.6 &\cellcolor[HTML]{f8f7fd}2.2 & &\cellcolor[HTML]{f5f4fd}2.9 &\cellcolor[HTML]{f1f0fc}3.9 &\cellcolor[HTML]{eeedfb}4.9 &\cellcolor[HTML]{eae8f9}6.1 &\cellcolor[HTML]{e7e5f9}6.9 &\cellcolor[HTML]{e5e3f8}7.4 \\
urd\_Arab &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fdfdff}0.8 &\cellcolor[HTML]{fefeff}0.4 &\cellcolor[HTML]{fefeff}0.5 &\cellcolor[HTML]{fefeff}0.3 &0.2 & &\cellcolor[HTML]{ebe9fa}5.7 &\cellcolor[HTML]{eae9fa}6.0 &\cellcolor[HTML]{e5e3f8}7.3 &\cellcolor[HTML]{e5e4f8}7.3 &\cellcolor[HTML]{eeecfb}5.0 &\cellcolor[HTML]{f3f3fc}3.3 \\
uzn\_Latn &\cellcolor[HTML]{dddaf6}9.8 &\cellcolor[HTML]{d8d5f4}11.3 &\cellcolor[HTML]{d5d2f4}11.9 &\cellcolor[HTML]{d8d5f4}11.3 &\cellcolor[HTML]{dedcf6}9.4 &\cellcolor[HTML]{e5e3f8}7.6 & &\cellcolor[HTML]{d3cff3}12.7 &\cellcolor[HTML]{c4c0ef}16.8 &\cellcolor[HTML]{b7b1eb}20.6 &\cellcolor[HTML]{aca6e8}23.7 &\cellcolor[HTML]{a8a1e7}24.9 &\cellcolor[HTML]{a59ee6}25.7 \\
vie\_Latn &0.1 &0.1 &0.1 &0.0 &0.0 &0.0 & &\cellcolor[HTML]{fefeff}0.3 &\cellcolor[HTML]{fefeff}0.3 &0.1 &0.1 &0.1 &0.0 \\
zsm\_Latn &\cellcolor[HTML]{f8f8fd}2.1 &\cellcolor[HTML]{f9f8fe}1.9 &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fdfdff}0.7 &\cellcolor[HTML]{fefeff}0.4 & &\cellcolor[HTML]{eae8f9}6.1 &\cellcolor[HTML]{e4e2f8}7.8 &\cellcolor[HTML]{e0def7}8.9 &\cellcolor[HTML]{dcdaf6}9.9 &\cellcolor[HTML]{dad8f5}10.4 &\cellcolor[HTML]{dfddf6}9.2 \\
\bottomrule
\end{tabular}

\begin{tabular}{lcccccccccccccc}\toprule
&\multicolumn{6}{c}{\textbf{Mistral Nemo}} & &\multicolumn{6}{c}{\textbf{Llama-2}} \\\cmidrule{2-7}\cmidrule{9-14}
\textbf{Lang.} &\textbf{1000} &\textbf{2000} &\textbf{4000} &\textbf{8000} &\textbf{16000} &\textbf{32000} & &\textbf{1000} &\textbf{2000} &\textbf{4000} &\textbf{8000} &\textbf{16000} &\textbf{32000} \\\midrule
afr\_Latn &\cellcolor[HTML]{fafafe}1.6 &\cellcolor[HTML]{f9f9fe}1.8 &\cellcolor[HTML]{faf9fe}1.7 &\cellcolor[HTML]{fbfafe}1.4 &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fcfcff}1.0 & &\cellcolor[HTML]{f5f4fd}3.0 &\cellcolor[HTML]{f1f0fb}4.1 &\cellcolor[HTML]{eeedfb}5.0 &\cellcolor[HTML]{eae9fa}6.0 &\cellcolor[HTML]{e8e6f9}6.6 &\cellcolor[HTML]{e8e6f9}6.7 \\
als\_Latn &\cellcolor[HTML]{f4f3fc}3.3 &\cellcolor[HTML]{f5f4fd}2.9 &\cellcolor[HTML]{f5f4fd}2.9 &\cellcolor[HTML]{f5f4fd}3.0 &\cellcolor[HTML]{f7f7fd}2.3 &\cellcolor[HTML]{faf9fe}1.6 & &\cellcolor[HTML]{e3e1f8}8.0 &\cellcolor[HTML]{dfdcf6}9.3 &\cellcolor[HTML]{dbd9f5}10.3 &\cellcolor[HTML]{d7d4f4}11.5 &\cellcolor[HTML]{dbd8f5}10.4 &\cellcolor[HTML]{dbd9f5}10.3 \\
arb\_Arab &0.2 &\cellcolor[HTML]{fffeff}0.3 &0.2 &\cellcolor[HTML]{fffeff}0.3 &\cellcolor[HTML]{fefeff}0.3 &\cellcolor[HTML]{fefeff}0.3 & &\cellcolor[HTML]{fafafe}1.6 &\cellcolor[HTML]{fdfdff}0.7 &\cellcolor[HTML]{fefeff}0.3 &0.1 &\cellcolor[HTML]{fefefe}-0.2 &\cellcolor[HTML]{fefefe}-0.2 \\
ary\_Arab &\cellcolor[HTML]{fefeff}0.4 &\cellcolor[HTML]{fefefe}-0.0 &0.0 &0.1 &0.1 &\cellcolor[HTML]{fefeff}0.3 & &\cellcolor[HTML]{fdfcff}0.8 &\cellcolor[HTML]{fefeff}0.4 &0.2 &0.1 &\cellcolor[HTML]{fefefe}-0.1 &\cellcolor[HTML]{fefefe}-0.2 \\
azj\_Latn &\cellcolor[HTML]{f3f2fc}3.6 &\cellcolor[HTML]{f0eefb}4.5 &\cellcolor[HTML]{edecfa}5.2 &\cellcolor[HTML]{ecebfa}5.5 &\cellcolor[HTML]{edecfa}5.1 &\cellcolor[HTML]{f0effb}4.3 & &\cellcolor[HTML]{d6d3f4}11.8 &\cellcolor[HTML]{cfcbf2}13.8 &\cellcolor[HTML]{c0bcee}18.0 &\cellcolor[HTML]{b8b3ec}20.4 &\cellcolor[HTML]{b1abea}22.4 &\cellcolor[HTML]{b2acea}22.1 \\
bel\_Cyrl &\cellcolor[HTML]{eae8fa}6.1 &\cellcolor[HTML]{e6e5f9}7.1 &\cellcolor[HTML]{e6e4f8}7.2 &\cellcolor[HTML]{e7e5f9}7.0 &\cellcolor[HTML]{eae9fa}6.1 &\cellcolor[HTML]{efeefb}4.6 & &\cellcolor[HTML]{faf9fe}1.7 &\cellcolor[HTML]{f6f6fd}2.6 &\cellcolor[HTML]{f5f4fc}3.1 &\cellcolor[HTML]{f1f0fc}4.0 &\cellcolor[HTML]{eeecfb}5.1 &\cellcolor[HTML]{eceafa}5.6 \\
ben\_Beng &\cellcolor[HTML]{fdfdff}0.7 &\cellcolor[HTML]{fefdff}0.6 &\cellcolor[HTML]{fdfdff}0.6 &\cellcolor[HTML]{fdfdff}0.6 &\cellcolor[HTML]{fdfdff}0.7 &\cellcolor[HTML]{fdfdff}0.7 & &\cellcolor[HTML]{fefeff}0.3 &0.1 &\cellcolor[HTML]{fefefd}-0.2 &\cellcolor[HTML]{fefdfd}-0.4 &\cellcolor[HTML]{fefdfc}-0.5 &\cellcolor[HTML]{fefcfb}-0.8 \\
bos\_Latn &\cellcolor[HTML]{f0effb}4.5 &\cellcolor[HTML]{eae9fa}6.0 &\cellcolor[HTML]{e9e8f9}6.3 &\cellcolor[HTML]{e9e7f9}6.5 &\cellcolor[HTML]{eae9fa}6.0 &\cellcolor[HTML]{ecebfa}5.4 & &\cellcolor[HTML]{e1dff7}8.6 &\cellcolor[HTML]{d6d3f4}11.9 &\cellcolor[HTML]{c9c5f0}15.6 &\cellcolor[HTML]{bcb7ed}19.2 &\cellcolor[HTML]{b0aae9}22.6 &\cellcolor[HTML]{a59ee6}25.7 \\
bul\_Cyrl &\cellcolor[HTML]{f8f7fd}2.2 &\cellcolor[HTML]{f7f6fd}2.5 &\cellcolor[HTML]{f7f6fd}2.5 &\cellcolor[HTML]{f7f6fd}2.5 &\cellcolor[HTML]{f8f7fd}2.2 &\cellcolor[HTML]{f9f8fe}1.9 & &\cellcolor[HTML]{f9f9fe}1.8 &\cellcolor[HTML]{f8f7fd}2.2 &\cellcolor[HTML]{f6f5fd}2.7 &\cellcolor[HTML]{f4f3fc}3.3 &\cellcolor[HTML]{f3f2fc}3.5 &\cellcolor[HTML]{f0effb}4.4 \\
cat\_Latn &\cellcolor[HTML]{faf9fe}1.7 &\cellcolor[HTML]{f8f8fe}2.0 &\cellcolor[HTML]{f9f9fe}1.7 &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{fbfafe}1.4 &\cellcolor[HTML]{fdfcff}0.8 & &\cellcolor[HTML]{f5f5fd}2.9 &\cellcolor[HTML]{f3f3fc}3.4 &\cellcolor[HTML]{f1f0fc}4.1 &\cellcolor[HTML]{f0effb}4.3 &\cellcolor[HTML]{efeefb}4.7 &\cellcolor[HTML]{eeedfb}4.8 \\
ces\_Latn &\cellcolor[HTML]{f6f6fd}2.6 &\cellcolor[HTML]{f2f1fc}3.8 &\cellcolor[HTML]{f0effb}4.5 &\cellcolor[HTML]{eeedfb}4.8 &\cellcolor[HTML]{edecfa}5.2 &\cellcolor[HTML]{f0effb}4.4 & &\cellcolor[HTML]{eeecfb}5.1 &\cellcolor[HTML]{e7e5f9}6.9 &\cellcolor[HTML]{e0ddf7}9.0 &\cellcolor[HTML]{d7d4f4}11.5 &\cellcolor[HTML]{d0cdf2}13.4 &\cellcolor[HTML]{cbc7f1}14.8 \\
cmn\_Hani &0.1 &0.1 &\cellcolor[HTML]{fefefe}-0.0 &0.1 &0.0 &0.0 & &0.0 &0.0 &0.0 &0.0 &\cellcolor[HTML]{fefdfd}-0.4 &\cellcolor[HTML]{fefdfc}-0.5 \\
dan\_Latn &\cellcolor[HTML]{fbfbfe}1.3 &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{faf9fe}1.6 &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{fafafe}1.6 &\cellcolor[HTML]{fafafe}1.5 & &\cellcolor[HTML]{f7f6fd}2.4 &\cellcolor[HTML]{f5f4fd}3.0 &\cellcolor[HTML]{f1f0fc}4.0 &\cellcolor[HTML]{efeefb}4.6 &\cellcolor[HTML]{eeedfb}5.0 &\cellcolor[HTML]{edebfa}5.4 \\
deu\_Latn &\cellcolor[HTML]{fdfdff}0.6 &\cellcolor[HTML]{fcfcff}0.9 &\cellcolor[HTML]{fcfbfe}1.1 &\cellcolor[HTML]{fbfbfe}1.1 &\cellcolor[HTML]{fbfbfe}1.3 &\cellcolor[HTML]{fcfbfe}1.1 & &\cellcolor[HTML]{faf9fe}1.6 &\cellcolor[HTML]{f8f7fd}2.1 &\cellcolor[HTML]{f5f4fd}2.9 &\cellcolor[HTML]{f3f2fc}3.6 &\cellcolor[HTML]{f0effb}4.4 &\cellcolor[HTML]{edecfb}5.1 \\
ekk\_Latn &\cellcolor[HTML]{f4f3fc}3.3 &\cellcolor[HTML]{f3f2fc}3.5 &\cellcolor[HTML]{f1f0fc}4.0 &\cellcolor[HTML]{efeefb}4.7 &\cellcolor[HTML]{efeefb}4.7 &\cellcolor[HTML]{f0effb}4.5 & &\cellcolor[HTML]{ebe9fa}5.9 &\cellcolor[HTML]{e6e4f8}7.2 &\cellcolor[HTML]{e2e0f7}8.3 &\cellcolor[HTML]{dddbf6}9.6 &\cellcolor[HTML]{dad7f5}10.7 &\cellcolor[HTML]{d4d1f3}12.4 \\
ell\_Grek &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{f8f8fe}2.0 &\cellcolor[HTML]{f7f6fd}2.5 &\cellcolor[HTML]{f6f5fd}2.8 &\cellcolor[HTML]{f7f7fd}2.4 &\cellcolor[HTML]{f9f9fe}1.7 & &\cellcolor[HTML]{fefeff}0.5 &\cellcolor[HTML]{fefefe}-0.2 &\cellcolor[HTML]{fefefd}-0.2 &\cellcolor[HTML]{fefefe}-0.2 &\cellcolor[HTML]{fefefe}-0.2 &0.0 \\
eus\_Latn &\cellcolor[HTML]{f1f0fb}4.2 &\cellcolor[HTML]{edecfa}5.1 &\cellcolor[HTML]{eceafa}5.7 &\cellcolor[HTML]{eae8fa}6.1 &\cellcolor[HTML]{ecebfa}5.4 &\cellcolor[HTML]{f0eefb}4.5 & &\cellcolor[HTML]{d4d1f4}12.3 &\cellcolor[HTML]{c8c4f0}15.7 &\cellcolor[HTML]{bdb8ed}19.0 &\cellcolor[HTML]{b5b0eb}21.1 &\cellcolor[HTML]{aea8e9}23.1 &\cellcolor[HTML]{a8a2e7}24.8 \\
fas\_Arab &\cellcolor[HTML]{fdfdff}0.8 &\cellcolor[HTML]{fdfdff}0.7 &\cellcolor[HTML]{fefeff}0.4 &\cellcolor[HTML]{fffeff}0.3 &0.2 &0.1 & &\cellcolor[HTML]{fdfcff}0.8 &\cellcolor[HTML]{fefeff}0.4 &\cellcolor[HTML]{fefeff}0.5 &\cellcolor[HTML]{fefefe}-0.0 &0.0 &0.1 \\
fin\_Latn &\cellcolor[HTML]{efeefb}4.7 &\cellcolor[HTML]{eceafa}5.6 &\cellcolor[HTML]{e5e4f8}7.4 &\cellcolor[HTML]{e3e1f8}8.2 &\cellcolor[HTML]{e1dff7}8.6 &\cellcolor[HTML]{e0ddf7}9.0 & &\cellcolor[HTML]{e4e3f8}7.7 &\cellcolor[HTML]{dbd8f5}10.5 &\cellcolor[HTML]{d1cdf3}13.3 &\cellcolor[HTML]{c7c3f0}16.0 &\cellcolor[HTML]{beb9ed}18.7 &\cellcolor[HTML]{b6b0eb}21.0 \\
fra\_Latn &\cellcolor[HTML]{fdfdff}0.7 &\cellcolor[HTML]{fdfdff}0.8 &\cellcolor[HTML]{fcfcff}0.9 &\cellcolor[HTML]{fdfdff}0.7 &\cellcolor[HTML]{fdfdff}0.7 &\cellcolor[HTML]{fefeff}0.5 & &\cellcolor[HTML]{f9f8fe}1.9 &\cellcolor[HTML]{f7f7fd}2.3 &\cellcolor[HTML]{f6f6fd}2.6 &\cellcolor[HTML]{f5f4fd}3.0 &\cellcolor[HTML]{f4f3fc}3.3 &\cellcolor[HTML]{f3f2fc}3.6 \\
glg\_Latn &\cellcolor[HTML]{fbfbfe}1.3 &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{fafafe}1.4 &\cellcolor[HTML]{fcfbfe}1.1 &\cellcolor[HTML]{fcfcff}1.0 &\cellcolor[HTML]{fdfdff}0.7 & &\cellcolor[HTML]{f6f5fd}2.7 &\cellcolor[HTML]{f4f4fc}3.1 &\cellcolor[HTML]{f2f1fc}3.8 &\cellcolor[HTML]{f0effb}4.3 &\cellcolor[HTML]{efeefb}4.7 &\cellcolor[HTML]{eeedfb}4.9 \\
guj\_Gujr &\cellcolor[HTML]{fcfcfe}1.0 &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fbfbfe}1.3 &\cellcolor[HTML]{fcfcfe}1.0 &\cellcolor[HTML]{fdfdff}0.8 & &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fdfdff}0.6 &\cellcolor[HTML]{fffeff}0.3 &0.2 &0.2 &\cellcolor[HTML]{fefefe}-0.0 \\
heb\_Hebr &\cellcolor[HTML]{dedbf6}9.6 &\cellcolor[HTML]{cfccf2}13.6 &\cellcolor[HTML]{c6c2f0}16.2 &\cellcolor[HTML]{c3bfef}17.1 &\cellcolor[HTML]{c9c5f0}15.6 &\cellcolor[HTML]{d1cef3}13.2 & &0.1 &0.0 &0.0 &\cellcolor[HTML]{fefefe}-0.1 &\cellcolor[HTML]{fefefe}-0.1 &\cellcolor[HTML]{fefefd}-0.3 \\
hin\_Deva &\cellcolor[HTML]{fefeff}0.4 &\cellcolor[HTML]{fefeff}0.4 &\cellcolor[HTML]{fefeff}0.5 &\cellcolor[HTML]{fefeff}0.3 &\cellcolor[HTML]{fffeff}0.3 &0.2 & &\cellcolor[HTML]{f6f5fd}2.6 &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fdfdff}0.6 &0.0 &0.0 &0.0 \\
hrv\_Latn &\cellcolor[HTML]{efeefb}4.6 &\cellcolor[HTML]{ebeafa}5.7 &\cellcolor[HTML]{e7e5f9}6.9 &\cellcolor[HTML]{e4e2f8}7.8 &\cellcolor[HTML]{e6e4f8}7.2 &\cellcolor[HTML]{eae9fa}6.0 & &\cellcolor[HTML]{dfddf6}9.3 &\cellcolor[HTML]{d2cff3}13.0 &\cellcolor[HTML]{c2beef}17.3 &\cellcolor[HTML]{b4aeeb}21.5 &\cellcolor[HTML]{a49de6}26.1 &\cellcolor[HTML]{968ee2}29.9 \\
hun\_Latn &\cellcolor[HTML]{f9f9fe}1.8 &\cellcolor[HTML]{f7f6fd}2.5 &\cellcolor[HTML]{f5f4fc}3.1 &\cellcolor[HTML]{f4f3fc}3.3 &\cellcolor[HTML]{f2f1fc}3.8 &\cellcolor[HTML]{f2f1fc}3.9 & &\cellcolor[HTML]{f1f0fc}4.0 &\cellcolor[HTML]{eceafa}5.7 &\cellcolor[HTML]{e5e3f8}7.4 &\cellcolor[HTML]{dfddf7}9.2 &\cellcolor[HTML]{d8d5f5}11.2 &\cellcolor[HTML]{d3d0f3}12.6 \\
hye\_Armn &0.3 &\cellcolor[HTML]{fefeff}0.3 &0.2 &0.3 &\cellcolor[HTML]{fffeff}0.3 &0.2 & &0.0 &0.2 &\cellcolor[HTML]{fefefe}-0.0 &\cellcolor[HTML]{fefefe}-0.1 &\cellcolor[HTML]{fefefe}-0.1 &\cellcolor[HTML]{fefefe}-0.1 \\
ind\_Latn &\cellcolor[HTML]{f9f9fe}1.8 &\cellcolor[HTML]{fafafe}1.6 &\cellcolor[HTML]{f9f9fe}1.7 &\cellcolor[HTML]{fcfbfe}1.1 &\cellcolor[HTML]{fdfcff}0.8 &\cellcolor[HTML]{fefeff}0.5 & &\cellcolor[HTML]{e5e4f8}7.4 &\cellcolor[HTML]{dfddf6}9.2 &\cellcolor[HTML]{d7d4f4}11.5 &\cellcolor[HTML]{d2cff3}12.9 &\cellcolor[HTML]{d8d5f4}11.3 &\cellcolor[HTML]{dddaf6}9.8 \\
isl\_Latn &\cellcolor[HTML]{f6f6fd}2.6 &\cellcolor[HTML]{f6f5fd}2.8 &\cellcolor[HTML]{f4f3fc}3.2 &\cellcolor[HTML]{f4f4fc}3.2 &\cellcolor[HTML]{f3f2fc}3.6 &\cellcolor[HTML]{f3f2fc}3.6 & &\cellcolor[HTML]{f4f4fc}3.2 &\cellcolor[HTML]{f1f0fb}4.2 &\cellcolor[HTML]{eeedfb}4.9 &\cellcolor[HTML]{ecebfa}5.4 &\cellcolor[HTML]{e8e6f9}6.8 &\cellcolor[HTML]{e5e3f8}7.5 \\
ita\_Latn &\cellcolor[HTML]{f5f4fd}3.0 &\cellcolor[HTML]{f4f3fc}3.3 &\cellcolor[HTML]{f4f3fc}3.4 &\cellcolor[HTML]{f5f4fc}3.1 &\cellcolor[HTML]{f8f7fd}2.2 &\cellcolor[HTML]{f9f8fe}1.9 & &\cellcolor[HTML]{eeedfb}4.8 &\cellcolor[HTML]{ebeafa}5.7 &\cellcolor[HTML]{e6e4f8}7.2 &\cellcolor[HTML]{e1dff7}8.6 &\cellcolor[HTML]{dddbf6}9.7 &\cellcolor[HTML]{d9d6f5}11.0 \\
jpn\_Jpan &\cellcolor[HTML]{fbfafe}1.4 &\cellcolor[HTML]{fafafe}1.4 &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{fafafe}1.6 &\cellcolor[HTML]{fbfbfe}1.3 & &0.0 &0.0 &0.0 &\cellcolor[HTML]{fefbfa}-1.0 &\cellcolor[HTML]{fef6f5}-2.2 &\cellcolor[HTML]{fdf4f2}-2.9 \\
kan\_Knda &0.2 &\cellcolor[HTML]{fefefe}-0.0 &\cellcolor[HTML]{fefeff}0.3 &0.2 &\cellcolor[HTML]{fefeff}0.4 &\cellcolor[HTML]{fefeff}0.4 & &\cellcolor[HTML]{fcfbfe}1.1 &\cellcolor[HTML]{fefeff}0.4 &0.2 &0.0 &0.1 &0.2 \\
kat\_Geor &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fcfbfe}1.1 &\cellcolor[HTML]{fcfcff}1.0 &\cellcolor[HTML]{fcfcfe}1.0 &\cellcolor[HTML]{fdfdff}0.7 &\cellcolor[HTML]{fefdff}0.6 & &\cellcolor[HTML]{fcfcff}0.9 &\cellcolor[HTML]{fefeff}0.4 &\cellcolor[HTML]{fffeff}0.3 &\cellcolor[HTML]{fefefe}-0.0 &\cellcolor[HTML]{fefdfd}-0.3 &\cellcolor[HTML]{fefefe}-0.1 \\
kaz\_Cyrl &\cellcolor[HTML]{e0def7}9.0 &\cellcolor[HTML]{dbd8f5}10.3 &\cellcolor[HTML]{dbd9f5}10.3 &\cellcolor[HTML]{e1dff7}8.6 &\cellcolor[HTML]{e7e6f9}6.9 &\cellcolor[HTML]{efeefb}4.7 & &\cellcolor[HTML]{c2beef}17.4 &\cellcolor[HTML]{b7b2eb}20.6 &\cellcolor[HTML]{aaa4e8}24.3 &\cellcolor[HTML]{aea8e9}23.1 &\cellcolor[HTML]{b9b4ec}20.0 &\cellcolor[HTML]{c3bfef}17.2 \\
khk\_Cyrl &\cellcolor[HTML]{e9e7f9}6.5 &\cellcolor[HTML]{e5e3f8}7.6 &\cellcolor[HTML]{e6e4f8}7.3 &\cellcolor[HTML]{eae9fa}6.0 &\cellcolor[HTML]{f0effb}4.3 &\cellcolor[HTML]{f5f4fc}3.1 & &\cellcolor[HTML]{d1cef3}13.1 &\cellcolor[HTML]{c9c5f0}15.5 &\cellcolor[HTML]{c2beef}17.4 &\cellcolor[HTML]{bdb9ed}18.8 &\cellcolor[HTML]{bdb8ed}19.0 &\cellcolor[HTML]{c0bbee}18.1 \\
khm\_Khmr &0.0 &0.1 &0.2 &0.2 &0.2 &0.2 & &0.0 &\cellcolor[HTML]{eeedfb}4.8 &\cellcolor[HTML]{fafafe}1.6 &\cellcolor[HTML]{fdfdff}0.7 &\cellcolor[HTML]{fefeff}0.3 &0.2 \\
kir\_Cyrl &\cellcolor[HTML]{dedcf6}9.4 &\cellcolor[HTML]{d8d6f5}11.1 &\cellcolor[HTML]{dad8f5}10.5 &\cellcolor[HTML]{dfddf7}9.1 &\cellcolor[HTML]{e6e4f8}7.3 &\cellcolor[HTML]{ecebfa}5.4 & &\cellcolor[HTML]{b6b1eb}20.8 &\cellcolor[HTML]{bab5ec}19.8 &\cellcolor[HTML]{aba5e8}24.0 &\cellcolor[HTML]{a49de6}26.0 &\cellcolor[HTML]{a39ce6}26.2 &\cellcolor[HTML]{a6a0e7}25.3 \\
kor\_Hang &0.0 &0.0 &0.1 &0.1 &0.1 &0.1 & &0.0 &0.0 &0.0 &0.1 &\cellcolor[HTML]{fefefd}-0.3 &\cellcolor[HTML]{fefefe}-0.2 \\
lit\_Latn &\cellcolor[HTML]{f3f2fc}3.5 &\cellcolor[HTML]{efeefb}4.7 &\cellcolor[HTML]{eeedfb}5.0 &\cellcolor[HTML]{eeecfb}5.1 &\cellcolor[HTML]{eeedfb}4.8 &\cellcolor[HTML]{f1f0fc}4.0 & &\cellcolor[HTML]{edecfa}5.1 &\cellcolor[HTML]{e5e3f8}7.5 &\cellcolor[HTML]{dedcf6}9.5 &\cellcolor[HTML]{d2cff3}12.8 &\cellcolor[HTML]{cbc7f1}15.0 &\cellcolor[HTML]{c5c1ef}16.6 \\
lvs\_Latn &\cellcolor[HTML]{f3f2fc}3.5 &\cellcolor[HTML]{f1f0fc}4.0 &\cellcolor[HTML]{f0effb}4.5 &\cellcolor[HTML]{efeefb}4.7 &\cellcolor[HTML]{f1f0fb}4.2 &\cellcolor[HTML]{f4f3fc}3.2 & &\cellcolor[HTML]{e9e7f9}6.4 &\cellcolor[HTML]{e4e2f8}7.8 &\cellcolor[HTML]{e2e0f7}8.3 &\cellcolor[HTML]{dddbf6}9.7 &\cellcolor[HTML]{d8d5f4}11.3 &\cellcolor[HTML]{d5d2f4}12.0 \\
mal\_Mlym &\cellcolor[HTML]{fefeff}0.3 &\cellcolor[HTML]{fefeff}0.4 &\cellcolor[HTML]{fefeff}0.5 &\cellcolor[HTML]{fefeff}0.5 &\cellcolor[HTML]{fefeff}0.5 &\cellcolor[HTML]{fefdff}0.5 & &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fefeff}0.4 &0.1 &0.1 &0.0 &\cellcolor[HTML]{fefefe}-0.0 \\
mar\_Deva &\cellcolor[HTML]{fcfcfe}1.1 &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{fbfbfe}1.3 &\cellcolor[HTML]{fafafe}1.4 &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fcfcff}0.9 & &\cellcolor[HTML]{fcfbfe}1.1 &\cellcolor[HTML]{fefeff}0.3 &0.1 &\cellcolor[HTML]{fefefe}-0.1 &\cellcolor[HTML]{fefdfd}-0.4 &\cellcolor[HTML]{fefcfb}-0.7 \\
mkd\_Cyrl &\cellcolor[HTML]{f8f8fe}2.0 &\cellcolor[HTML]{f8f7fd}2.2 &\cellcolor[HTML]{f7f6fd}2.4 &\cellcolor[HTML]{f6f5fd}2.7 &\cellcolor[HTML]{f7f6fd}2.4 &\cellcolor[HTML]{f9f8fe}1.9 & &\cellcolor[HTML]{f8f7fd}2.1 &\cellcolor[HTML]{f7f6fd}2.5 &\cellcolor[HTML]{f6f6fd}2.6 &\cellcolor[HTML]{f4f4fc}3.1 &\cellcolor[HTML]{f2f1fc}3.8 &\cellcolor[HTML]{f1f0fb}4.2 \\
mya\_Mymr &\cellcolor[HTML]{e2e0f7}8.2 &\cellcolor[HTML]{d9d6f5}11.0 &\cellcolor[HTML]{ccc8f1}14.6 &\cellcolor[HTML]{c1bdee}17.6 &\cellcolor[HTML]{b3aeea}21.6 &\cellcolor[HTML]{aca6e8}23.7 & &\cellcolor[HTML]{ebe9fa}5.9 &\cellcolor[HTML]{f7f7fd}2.3 &\cellcolor[HTML]{fcfcff}1.0 &\cellcolor[HTML]{fefefd}-0.3 &\cellcolor[HTML]{fefcfc}-0.7 &\cellcolor[HTML]{fef8f7}-1.6 \\
nld\_Latn &\cellcolor[HTML]{fbfbfe}1.3 &\cellcolor[HTML]{f9f9fe}1.7 &\cellcolor[HTML]{f9f9fe}1.7 &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{fbfafe}1.3 &\cellcolor[HTML]{fcfcff}1.0 & &\cellcolor[HTML]{f4f3fc}3.3 &\cellcolor[HTML]{f1f0fc}4.0 &\cellcolor[HTML]{eeecfb}5.1 &\cellcolor[HTML]{e8e7f9}6.6 &\cellcolor[HTML]{e7e5f9}7.1 &\cellcolor[HTML]{e7e5f9}7.0 \\
nno\_Latn &\cellcolor[HTML]{fafafe}1.6 &\cellcolor[HTML]{f9f9fe}1.7 &\cellcolor[HTML]{f9f9fe}1.7 &\cellcolor[HTML]{faf9fe}1.6 &\cellcolor[HTML]{faf9fe}1.6 &\cellcolor[HTML]{fafafe}1.4 & &\cellcolor[HTML]{f3f2fc}3.5 &\cellcolor[HTML]{f2f1fc}3.9 &\cellcolor[HTML]{f0effb}4.5 &\cellcolor[HTML]{edecfa}5.2 &\cellcolor[HTML]{ebeafa}5.8 &\cellcolor[HTML]{edecfa}5.3 \\
nob\_Latn &\cellcolor[HTML]{fcfcff}0.9 &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fbfbfe}1.1 &\cellcolor[HTML]{fbfbfe}1.3 &\cellcolor[HTML]{fbfafe}1.3 &\cellcolor[HTML]{fbfbfe}1.3 & &\cellcolor[HTML]{f8f7fd}2.2 &\cellcolor[HTML]{f6f5fd}2.7 &\cellcolor[HTML]{f3f2fc}3.5 &\cellcolor[HTML]{f1f0fb}4.2 &\cellcolor[HTML]{f0effb}4.5 &\cellcolor[HTML]{eeedfb}5.0 \\
npi\_Deva &\cellcolor[HTML]{faf9fe}1.6 &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{f9f9fe}1.7 &\cellcolor[HTML]{f9f9fe}1.7 &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{fbfafe}1.3 & &\cellcolor[HTML]{fefeff}0.5 &\cellcolor[HTML]{fefdff}0.6 &0.2 &0.0 &\cellcolor[HTML]{fefefe}-0.1 &\cellcolor[HTML]{fefefe}-0.1 \\
ory\_Orya &0.0 &0.0 &0.0 &0.0 &0.0 &0.0 & &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fdfdff}0.6 &\cellcolor[HTML]{fefeff}0.3 &0.1 &\cellcolor[HTML]{fefefe}-0.0 &0.0 \\
pan\_Guru &0.1 &0.0 &0.1 &\cellcolor[HTML]{fefefe}-0.0 &0.0 &0.0 & &\cellcolor[HTML]{fcfcfe}1.1 &\cellcolor[HTML]{fefeff}0.4 &0.1 &\cellcolor[HTML]{fefefe}-0.0 &\cellcolor[HTML]{fefefe}-0.0 &\cellcolor[HTML]{fefefe}-0.1 \\
pol\_Latn &\cellcolor[HTML]{f2f1fc}3.8 &\cellcolor[HTML]{eeecfb}5.1 &\cellcolor[HTML]{eae8fa}6.2 &\cellcolor[HTML]{e6e4f8}7.2 &\cellcolor[HTML]{e6e4f8}7.3 &\cellcolor[HTML]{e8e6f9}6.7 & &\cellcolor[HTML]{e5e3f8}7.5 &\cellcolor[HTML]{d9d6f5}11.0 &\cellcolor[HTML]{cbc7f1}14.9 &\cellcolor[HTML]{bcb7ed}19.3 &\cellcolor[HTML]{b0abea}22.4 &\cellcolor[HTML]{a49ee6}25.9 \\
por\_Latn &\cellcolor[HTML]{fcfbfe}1.1 &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fcfbfe}1.1 &\cellcolor[HTML]{fcfcff}1.0 &\cellcolor[HTML]{fdfcff}0.8 &\cellcolor[HTML]{fefeff}0.5 & &\cellcolor[HTML]{f9f8fe}1.9 &\cellcolor[HTML]{f6f6fd}2.6 &\cellcolor[HTML]{f4f3fc}3.3 &\cellcolor[HTML]{f2f1fc}3.8 &\cellcolor[HTML]{f1f0fb}4.1 &\cellcolor[HTML]{f0effb}4.3 \\
ron\_Latn &\cellcolor[HTML]{f3f2fc}3.5 &\cellcolor[HTML]{f1f0fc}4.1 &\cellcolor[HTML]{f1f0fb}4.2 &\cellcolor[HTML]{f0effb}4.3 &\cellcolor[HTML]{f2f2fc}3.7 &\cellcolor[HTML]{f7f6fd}2.5 & &\cellcolor[HTML]{e8e6f9}6.8 &\cellcolor[HTML]{e4e2f8}7.7 &\cellcolor[HTML]{dfddf7}9.1 &\cellcolor[HTML]{dad7f5}10.7 &\cellcolor[HTML]{d6d3f4}11.7 &\cellcolor[HTML]{d6d3f4}11.7 \\
rus\_Cyrl &\cellcolor[HTML]{f9f8fe}1.9 &\cellcolor[HTML]{f8f7fd}2.2 &\cellcolor[HTML]{f6f6fd}2.6 &\cellcolor[HTML]{f6f5fd}2.7 &\cellcolor[HTML]{f6f5fd}2.8 &\cellcolor[HTML]{f7f6fd}2.4 & &\cellcolor[HTML]{fbfafe}1.4 &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{f9f8fe}2.0 &\cellcolor[HTML]{f7f6fd}2.5 &\cellcolor[HTML]{f6f6fd}2.6 &\cellcolor[HTML]{f4f4fc}3.2 \\
sin\_Sinh &0.0 &0.0 &0.0 &0.1 &0.0 &0.0 & &\cellcolor[HTML]{fcfcff}1.0 &\cellcolor[HTML]{fefeff}0.3 &0.2 &0.1 &\cellcolor[HTML]{fefefe}-0.0 &0.0 \\
slk\_Latn &\cellcolor[HTML]{f0eefb}4.5 &\cellcolor[HTML]{eeecfb}5.1 &\cellcolor[HTML]{edebfa}5.4 &\cellcolor[HTML]{ebeafa}5.8 &\cellcolor[HTML]{ebeafa}5.7 &\cellcolor[HTML]{edecfb}5.1 & &\cellcolor[HTML]{e9e8f9}6.3 &\cellcolor[HTML]{e2e0f7}8.4 &\cellcolor[HTML]{d9d6f5}10.9 &\cellcolor[HTML]{d2cff3}13.0 &\cellcolor[HTML]{cac6f1}15.2 &\cellcolor[HTML]{c8c4f0}15.8 \\
slv\_Latn &\cellcolor[HTML]{f0eefb}4.5 &\cellcolor[HTML]{ebeafa}5.7 &\cellcolor[HTML]{e9e7f9}6.4 &\cellcolor[HTML]{e8e6f9}6.6 &\cellcolor[HTML]{eae8f9}6.2 &\cellcolor[HTML]{edecfa}5.3 & &\cellcolor[HTML]{e2e0f7}8.3 &\cellcolor[HTML]{d9d7f5}10.9 &\cellcolor[HTML]{cecbf2}13.9 &\cellcolor[HTML]{c3bfef}17.1 &\cellcolor[HTML]{bbb7ed}19.3 &\cellcolor[HTML]{b5afeb}21.3 \\
spa\_Latn &\cellcolor[HTML]{fafafe}1.4 &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{fbfafe}1.4 &\cellcolor[HTML]{fbfafe}1.4 &\cellcolor[HTML]{fcfcff}1.0 &\cellcolor[HTML]{fdfdff}0.7 & &\cellcolor[HTML]{f5f4fd}2.9 &\cellcolor[HTML]{f3f2fc}3.5 &\cellcolor[HTML]{f2f1fc}3.9 &\cellcolor[HTML]{f0effb}4.4 &\cellcolor[HTML]{eeecfb}5.1 &\cellcolor[HTML]{edebfa}5.4 \\
srp\_Cyrl &\cellcolor[HTML]{ecebfa}5.5 &\cellcolor[HTML]{e8e6f9}6.6 &\cellcolor[HTML]{e4e2f8}7.9 &\cellcolor[HTML]{e3e1f8}8.0 &\cellcolor[HTML]{e3e1f8}8.0 &\cellcolor[HTML]{e8e6f9}6.6 & &\cellcolor[HTML]{faf9fe}1.6 &\cellcolor[HTML]{f9f9fe}1.8 &\cellcolor[HTML]{f8f7fd}2.3 &\cellcolor[HTML]{f6f5fd}2.8 &\cellcolor[HTML]{f3f2fc}3.6 &\cellcolor[HTML]{f1f0fc}4.0 \\
swe\_Latn &\cellcolor[HTML]{fdfdff}0.7 &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fbfbfe}1.2 &\cellcolor[HTML]{fafafe}1.6 &\cellcolor[HTML]{faf9fe}1.6 &\cellcolor[HTML]{fbfafe}1.3 & &\cellcolor[HTML]{f9f8fe}1.9 &\cellcolor[HTML]{f6f6fd}2.6 &\cellcolor[HTML]{f4f3fc}3.3 &\cellcolor[HTML]{f0effb}4.3 &\cellcolor[HTML]{f0effb}4.5 &\cellcolor[HTML]{f0effb}4.5 \\
swh\_Latn &\cellcolor[HTML]{e3e1f8}8.1 &\cellcolor[HTML]{dfddf7}9.1 &\cellcolor[HTML]{dfdcf6}9.3 &\cellcolor[HTML]{dedbf6}9.6 &\cellcolor[HTML]{e2e0f7}8.4 &\cellcolor[HTML]{e8e6f9}6.7 & &\cellcolor[HTML]{b7b2eb}20.6 &\cellcolor[HTML]{a09ae5}27.0 &\cellcolor[HTML]{867ede}34.4 &\cellcolor[HTML]{7268d8}41.5 &\cellcolor[HTML]{7268d8}43.7 &\cellcolor[HTML]{7268d8}44.4 \\
tam\_Taml &\cellcolor[HTML]{fefeff}0.3 &\cellcolor[HTML]{fefeff}0.3 &\cellcolor[HTML]{fefdff}0.6 &\cellcolor[HTML]{fefeff}0.4 &\cellcolor[HTML]{fefeff}0.5 &\cellcolor[HTML]{fefeff}0.4 & &\cellcolor[HTML]{f0effb}4.3 &\cellcolor[HTML]{fbfafe}1.4 &\cellcolor[HTML]{fdfdff}0.6 &0.1 &\cellcolor[HTML]{fefefe}-0.1 &\cellcolor[HTML]{fefefd}-0.3 \\
tel\_Telu &\cellcolor[HTML]{fcfbfe}1.1 &\cellcolor[HTML]{fbfafe}1.4 &\cellcolor[HTML]{f9f8fe}1.9 &\cellcolor[HTML]{faf9fe}1.6 &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{fbfbfe}1.2 & &\cellcolor[HTML]{faf9fe}1.6 &\cellcolor[HTML]{fefeff}0.5 &\cellcolor[HTML]{fefeff}0.5 &0.2 &\cellcolor[HTML]{fefefe}-0.0 &\cellcolor[HTML]{fefdfd}-0.3 \\
tha\_Thai &\cellcolor[HTML]{8178dd}35.8 &\cellcolor[HTML]{7268d8}53.6 &\cellcolor[HTML]{7268d8}70.2 &\cellcolor[HTML]{7268d8}86.2 &\cellcolor[HTML]{7268d8}100.4 &\cellcolor[HTML]{7268d8}110.9 & &\cellcolor[HTML]{dedbf6}9.6 &\cellcolor[HTML]{f2f1fc}3.9 &\cellcolor[HTML]{fafafe}1.5 &\cellcolor[HTML]{fefeff}0.5 &0.2 &0.3 \\
tur\_Latn &\cellcolor[HTML]{f8f7fd}2.1 &\cellcolor[HTML]{f5f5fd}2.9 &\cellcolor[HTML]{f4f3fc}3.4 &\cellcolor[HTML]{f3f2fc}3.5 &\cellcolor[HTML]{f3f2fc}3.6 &\cellcolor[HTML]{f5f4fd}3.0 & &\cellcolor[HTML]{e0def7}8.9 &\cellcolor[HTML]{dbd8f5}10.4 &\cellcolor[HTML]{d4d1f3}12.4 &\cellcolor[HTML]{cdc9f1}14.4 &\cellcolor[HTML]{c8c4f0}15.8 &\cellcolor[HTML]{cac6f1}15.2 \\
ukr\_Cyrl &\cellcolor[HTML]{f1f0fc}4.0 &\cellcolor[HTML]{efedfb}4.8 &\cellcolor[HTML]{eeecfb}5.1 &\cellcolor[HTML]{edecfa}5.3 &\cellcolor[HTML]{edecfa}5.2 &\cellcolor[HTML]{f0effb}4.5 & &\cellcolor[HTML]{f9f9fe}1.8 &\cellcolor[HTML]{f8f8fd}2.1 &\cellcolor[HTML]{f7f6fd}2.5 &\cellcolor[HTML]{f4f3fc}3.3 &\cellcolor[HTML]{f2f1fc}3.8 &\cellcolor[HTML]{f1f0fc}4.0 \\
urd\_Arab &\cellcolor[HTML]{fcfcff}0.9 &\cellcolor[HTML]{fcfcff}1.0 &\cellcolor[HTML]{fdfdff}0.6 &\cellcolor[HTML]{fefeff}0.5 &\cellcolor[HTML]{fefeff}0.3 &0.1 & &\cellcolor[HTML]{fdfdff}0.6 &0.2 &\cellcolor[HTML]{fefefe}-0.1 &\cellcolor[HTML]{fefefe}-0.1 &\cellcolor[HTML]{fefefe}-0.1 &\cellcolor[HTML]{fefefe}-0.0 \\
uzn\_Latn &\cellcolor[HTML]{e6e4f9}7.2 &\cellcolor[HTML]{e4e2f8}7.9 &\cellcolor[HTML]{e3e0f7}8.2 &\cellcolor[HTML]{e3e1f8}8.1 &\cellcolor[HTML]{e5e3f8}7.5 &\cellcolor[HTML]{e9e8f9}6.3 & &\cellcolor[HTML]{cfccf2}13.6 &\cellcolor[HTML]{c3bfef}17.1 &\cellcolor[HTML]{b5b0eb}21.1 &\cellcolor[HTML]{afa9e9}22.9 &\cellcolor[HTML]{aca6e8}23.6 &\cellcolor[HTML]{b3adea}21.8 \\
vie\_Latn &0.1 &0.1 &0.0 &0.0 &0.0 &0.0 & &\cellcolor[HTML]{dfdcf6}9.3 &\cellcolor[HTML]{f1f0fc}4.1 &\cellcolor[HTML]{f7f6fd}2.5 &\cellcolor[HTML]{f9f9fe}1.8 &\cellcolor[HTML]{fcfcff}0.9 &0.2 \\
zsm\_Latn &\cellcolor[HTML]{f8f7fd}2.2 &\cellcolor[HTML]{f9f8fe}2.0 &\cellcolor[HTML]{fbfafe}1.4 &\cellcolor[HTML]{fcfcff}1.0 &\cellcolor[HTML]{fefeff}0.5 &\cellcolor[HTML]{fefeff}0.4 & &\cellcolor[HTML]{e0ddf7}9.0 &\cellcolor[HTML]{dbd9f6}10.2 &\cellcolor[HTML]{d6d3f4}11.8 &\cellcolor[HTML]{d8d5f4}11.3 &\cellcolor[HTML]{d7d4f4}11.5 &\cellcolor[HTML]{e0ddf7}9.0 \\
\bottomrule
\end{tabular}

\begin{tabular}{lcccccccccccccc}\toprule
&\multicolumn{6}{c}{\textbf{Llama-3 extension}} & &\multicolumn{6}{c}{\textbf{Qwen-2.5 extension}} \\\cmidrule{2-7}\cmidrule{9-14}
\textbf{Lang.} &\textbf{1000} &\textbf{2000} &\textbf{4000} &\textbf{8000} &\textbf{16000} &\textbf{32000} & &\textbf{1000} &\textbf{2000} &\textbf{4000} &\textbf{8000} &\textbf{16000} &\textbf{32000} \\\midrule
afr\_Latn &\cellcolor[HTML]{fefefe}-74 &\cellcolor[HTML]{fdfdfe}-156 &\cellcolor[HTML]{fafafd}-357 &\cellcolor[HTML]{f4f4fc}-875 &\cellcolor[HTML]{e9e7f9}-1858 &\cellcolor[HTML]{d4d1f3}-3692 & &\cellcolor[HTML]{fefefe}-72 &\cellcolor[HTML]{fdfdfe}-155 &\cellcolor[HTML]{fafafd}-350 &\cellcolor[HTML]{f4f4fc}-882 &\cellcolor[HTML]{e9e7f9}-1859 &\cellcolor[HTML]{d4d1f3}-3676 \\
als\_Latn &\cellcolor[HTML]{fdfdfe}-95 &\cellcolor[HTML]{fcfcfe}-229 &\cellcolor[HTML]{f9f9fd}-478 &\cellcolor[HTML]{f3f2fb}-983 &\cellcolor[HTML]{e7e6f8}-2004 &\cellcolor[HTML]{cfccf1}-4071 & &\cellcolor[HTML]{fdfdfe}-94 &\cellcolor[HTML]{fcfcfe}-231 &\cellcolor[HTML]{f9f8fd}-491 &\cellcolor[HTML]{f3f2fb}-1007 &\cellcolor[HTML]{e7e5f8}-2045 &\cellcolor[HTML]{cfcbf1}-4127 \\
arb\_Arab &\cellcolor[HTML]{fefefe}-23 &\cellcolor[HTML]{fefefe}-58 &\cellcolor[HTML]{fdfdfe}-141 &\cellcolor[HTML]{fbfbfe}-293 &\cellcolor[HTML]{f8f8fd}-547 &\cellcolor[HTML]{f2f2fb}-1039 & &\cellcolor[HTML]{fefefe}-15 &\cellcolor[HTML]{fefefe}-52 &\cellcolor[HTML]{fdfcfe}-162 &\cellcolor[HTML]{fafafd}-366 &\cellcolor[HTML]{f6f5fc}-733 &\cellcolor[HTML]{eeedfa}-1442 \\
ary\_Arab &\cellcolor[HTML]{fefefe}-27 &\cellcolor[HTML]{fefefe}-52 &\cellcolor[HTML]{fdfdfe}-120 &\cellcolor[HTML]{fbfbfe}-264 &\cellcolor[HTML]{f9f8fd}-510 &\cellcolor[HTML]{f3f3fb}-954 & &\cellcolor[HTML]{fefefe}-28 &\cellcolor[HTML]{fefefe}-63 &\cellcolor[HTML]{fdfcfe}-163 &\cellcolor[HTML]{fafafd}-358 &\cellcolor[HTML]{f6f5fc}-728 &\cellcolor[HTML]{eeedfa}-1411 \\
azj\_Latn &\cellcolor[HTML]{fdfdfe}-161 &\cellcolor[HTML]{fafafd}-359 &\cellcolor[HTML]{f5f4fc}-851 &\cellcolor[HTML]{ebe9f9}-1714 &\cellcolor[HTML]{d3d0f2}-3746 &\cellcolor[HTML]{a59ee6}-7746 & &\cellcolor[HTML]{fdfdfe}-137 &\cellcolor[HTML]{fbfbfe}-309 &\cellcolor[HTML]{f6f5fc}-751 &\cellcolor[HTML]{ecebf9}-1591 &\cellcolor[HTML]{d6d3f3}-3525 &\cellcolor[HTML]{a9a3e7}-7402 \\
bel\_Cyrl &\cellcolor[HTML]{fefefe}-54 &\cellcolor[HTML]{fdfdfe}-132 &\cellcolor[HTML]{fbfbfd}-313 &\cellcolor[HTML]{f6f6fc}-699 &\cellcolor[HTML]{edecfa}-1509 &\cellcolor[HTML]{d9d7f4}-3216 & &\cellcolor[HTML]{fefefe}-22 &\cellcolor[HTML]{fdfdfe}-89 &\cellcolor[HTML]{fcfcfe}-241 &\cellcolor[HTML]{f8f7fd}-596 &\cellcolor[HTML]{efeefa}-1330 &\cellcolor[HTML]{dedcf6}-2772 \\
ben\_Beng &0 &0 &0 &0 &\cellcolor[HTML]{fefefe}-46 &\cellcolor[HTML]{fcfcfe}-225 & &0 &0 &0 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-53 &\cellcolor[HTML]{fcfcfe}-231 \\
bos\_Latn &\cellcolor[HTML]{fefefe}-62 &\cellcolor[HTML]{fcfcfe}-174 &\cellcolor[HTML]{f8f8fd}-520 &\cellcolor[HTML]{eeedfa}-1449 &\cellcolor[HTML]{d5d2f3}-3599 &\cellcolor[HTML]{9c95e3}-8484 & &\cellcolor[HTML]{fefefe}-67 &\cellcolor[HTML]{fcfcfe}-184 &\cellcolor[HTML]{f8f7fd}-567 &\cellcolor[HTML]{edebfa}-1534 &\cellcolor[HTML]{d3d0f3}-3737 &\cellcolor[HTML]{9a92e3}-8707 \\
bul\_Cyrl &\cellcolor[HTML]{fdfdfe}-89 &\cellcolor[HTML]{fcfcfe}-198 &\cellcolor[HTML]{f9f8fd}-516 &\cellcolor[HTML]{efeefa}-1312 &\cellcolor[HTML]{d7d5f4}-3364 &\cellcolor[HTML]{a49ee5}-7808 & &\cellcolor[HTML]{fdfdfe}-111 &\cellcolor[HTML]{fbfbfe}-274 &\cellcolor[HTML]{f6f6fc}-704 &\cellcolor[HTML]{eae9f9}-1751 &\cellcolor[HTML]{cecbf1}-4177 &\cellcolor[HTML]{9790e2}-8923 \\
cat\_Latn &\cellcolor[HTML]{fefefe}-60 &\cellcolor[HTML]{fdfdfe}-133 &\cellcolor[HTML]{fbfafd}-342 &\cellcolor[HTML]{f6f5fc}-752 &\cellcolor[HTML]{eceaf9}-1635 &\cellcolor[HTML]{d7d4f3}-3432 & &\cellcolor[HTML]{fefefe}-60 &\cellcolor[HTML]{fdfdfe}-132 &\cellcolor[HTML]{fbfafd}-337 &\cellcolor[HTML]{f6f5fc}-748 &\cellcolor[HTML]{eceaf9}-1623 &\cellcolor[HTML]{d7d4f4}-3404 \\
ces\_Latn &\cellcolor[HTML]{fdfdfe}-118 &\cellcolor[HTML]{fcfcfe}-183 &\cellcolor[HTML]{fafafd}-377 &\cellcolor[HTML]{f4f3fc}-890 &\cellcolor[HTML]{e7e5f8}-2052 &\cellcolor[HTML]{c9c5f0}-4599 & &\cellcolor[HTML]{fefefe}-64 &\cellcolor[HTML]{fdfcfe}-166 &\cellcolor[HTML]{fafafd}-392 &\cellcolor[HTML]{f3f3fb}-951 &\cellcolor[HTML]{e4e2f7}-2273 &\cellcolor[HTML]{c4c0ee}-5015 \\
cmn\_Hani &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-3 &\cellcolor[HTML]{fefefe}-5 &\cellcolor[HTML]{fefefe}-15 &\cellcolor[HTML]{fefefe}-25 &\cellcolor[HTML]{fefefe}-74 & &\cellcolor[HTML]{fefefe}-16 &\cellcolor[HTML]{fefefe}-36 &\cellcolor[HTML]{fefefe}-66 &\cellcolor[HTML]{fdfdfe}-122 &\cellcolor[HTML]{fcfcfe}-211 &\cellcolor[HTML]{fafafd}-369 \\
dan\_Latn &\cellcolor[HTML]{fefefe}-55 &\cellcolor[HTML]{fdfdfe}-141 &\cellcolor[HTML]{fbfafd}-326 &\cellcolor[HTML]{f6f5fc}-747 &\cellcolor[HTML]{ecebf9}-1578 &\cellcolor[HTML]{d6d3f3}-3522 & &\cellcolor[HTML]{fefefe}-53 &\cellcolor[HTML]{fdfdfe}-140 &\cellcolor[HTML]{fbfafd}-323 &\cellcolor[HTML]{f6f5fc}-752 &\cellcolor[HTML]{ecebf9}-1585 &\cellcolor[HTML]{d6d3f3}-3516 \\
deu\_Latn &\cellcolor[HTML]{fefefe}-15 &\cellcolor[HTML]{fefefe}-67 &\cellcolor[HTML]{fcfcfe}-239 &\cellcolor[HTML]{f7f6fc}-659 &\cellcolor[HTML]{edecfa}-1511 &\cellcolor[HTML]{d8d5f4}-3362 & &\cellcolor[HTML]{fefefe}-16 &\cellcolor[HTML]{fefefe}-70 &\cellcolor[HTML]{fcfbfe}-247 &\cellcolor[HTML]{f7f6fc}-666 &\cellcolor[HTML]{edecfa}-1524 &\cellcolor[HTML]{d7d5f4}-3375 \\
ekk\_Latn &\cellcolor[HTML]{fefefe}-78 &\cellcolor[HTML]{fcfcfe}-175 &\cellcolor[HTML]{fafafd}-359 &\cellcolor[HTML]{f4f3fc}-899 &\cellcolor[HTML]{e6e5f8}-2084 &\cellcolor[HTML]{c9c5f0}-4612 & &\cellcolor[HTML]{fefefe}-77 &\cellcolor[HTML]{fcfcfe}-177 &\cellcolor[HTML]{fafafd}-361 &\cellcolor[HTML]{f4f3fc}-897 &\cellcolor[HTML]{e6e5f8}-2092 &\cellcolor[HTML]{c9c5f0}-4628 \\
ell\_Grek &\cellcolor[HTML]{fefefe}-23 &\cellcolor[HTML]{fefefe}-59 &\cellcolor[HTML]{fdfdfe}-148 &\cellcolor[HTML]{fbfafd}-324 &\cellcolor[HTML]{f7f6fc}-680 &\cellcolor[HTML]{f0effb}-1230 & &0 &\cellcolor[HTML]{fefefe}-4 &\cellcolor[HTML]{fefefe}-13 &\cellcolor[HTML]{fefefe}-31 &\cellcolor[HTML]{fefefe}-73 &\cellcolor[HTML]{fefefe}-19 \\
eus\_Latn &\cellcolor[HTML]{fdfdfe}-115 &\cellcolor[HTML]{fbfbfe}-302 &\cellcolor[HTML]{f7f6fc}-682 &\cellcolor[HTML]{ecebf9}-1604 &\cellcolor[HTML]{d6d3f3}-3492 &\cellcolor[HTML]{a8a2e6}-7483 & &\cellcolor[HTML]{fdfdfe}-111 &\cellcolor[HTML]{fbfbfe}-299 &\cellcolor[HTML]{f7f6fc}-689 &\cellcolor[HTML]{eceaf9}-1631 &\cellcolor[HTML]{d6d3f3}-3533 &\cellcolor[HTML]{a7a1e6}-7564 \\
fas\_Arab &\cellcolor[HTML]{fefefe}-27 &\cellcolor[HTML]{fefefe}-64 &\cellcolor[HTML]{fdfdfe}-138 &\cellcolor[HTML]{fbfbfe}-286 &\cellcolor[HTML]{f8f8fd}-562 &\cellcolor[HTML]{f2f1fb}-1070 & &\cellcolor[HTML]{fefefe}-38 &\cellcolor[HTML]{fdfdfe}-116 &\cellcolor[HTML]{fbfbfe}-261 &\cellcolor[HTML]{f8f7fd}-582 &\cellcolor[HTML]{f2f1fb}-1121 &\cellcolor[HTML]{e6e4f8}-2116 \\
fin\_Latn &\cellcolor[HTML]{fefefe}-57 &\cellcolor[HTML]{fcfcfe}-184 &\cellcolor[HTML]{f9f9fd}-439 &\cellcolor[HTML]{f2f1fb}-1097 &\cellcolor[HTML]{e1dff6}-2525 &\cellcolor[HTML]{bbb6ec}-5813 & &\cellcolor[HTML]{fefefe}-58 &\cellcolor[HTML]{fcfcfe}-186 &\cellcolor[HTML]{f9f9fd}-443 &\cellcolor[HTML]{f2f1fb}-1105 &\cellcolor[HTML]{e1dff6}-2536 &\cellcolor[HTML]{bbb6ec}-5844 \\
fra\_Latn &\cellcolor[HTML]{fefefe}-17 &\cellcolor[HTML]{fefefe}-57 &\cellcolor[HTML]{fcfcfe}-178 &\cellcolor[HTML]{f9f8fd}-489 &\cellcolor[HTML]{f1f0fb}-1153 &\cellcolor[HTML]{e1dff6}-2505 & &\cellcolor[HTML]{fefefe}-17 &\cellcolor[HTML]{fefefe}-57 &\cellcolor[HTML]{fdfcfe}-171 &\cellcolor[HTML]{f9f9fd}-476 &\cellcolor[HTML]{f1f0fb}-1142 &\cellcolor[HTML]{e2e0f7}-2488 \\
glg\_Latn &\cellcolor[HTML]{fefefe}-36 &\cellcolor[HTML]{fdfdfe}-100 &\cellcolor[HTML]{fcfcfe}-216 &\cellcolor[HTML]{f9f8fd}-501 &\cellcolor[HTML]{f1f0fb}-1147 &\cellcolor[HTML]{e0ddf6}-2670 & &\cellcolor[HTML]{fefefe}-36 &\cellcolor[HTML]{fdfdfe}-105 &\cellcolor[HTML]{fcfcfe}-227 &\cellcolor[HTML]{f9f8fd}-497 &\cellcolor[HTML]{f1f0fb}-1147 &\cellcolor[HTML]{e0ddf6}-2663 \\
guj\_Gujr &0 &0 &\cellcolor[HTML]{fefefe}-4 &\cellcolor[HTML]{fefefe}-14 &\cellcolor[HTML]{fefefe}-73 &\cellcolor[HTML]{fbfbfe}-273 & &0 &0 &\cellcolor[HTML]{fefefe}-4 &\cellcolor[HTML]{fefefe}-14 &\cellcolor[HTML]{fefefe}-64 &\cellcolor[HTML]{fbfbfe}-271 \\
heb\_Hebr &\cellcolor[HTML]{fdfdfe}-133 &\cellcolor[HTML]{fbfbfe}-310 &\cellcolor[HTML]{f7f6fc}-672 &\cellcolor[HTML]{edecfa}-1480 &\cellcolor[HTML]{d9d6f4}-3254 &\cellcolor[HTML]{bcb7ec}-5762 & &\cellcolor[HTML]{fefefe}-36 &\cellcolor[HTML]{fdfdfe}-132 &\cellcolor[HTML]{f9f9fd}-482 &\cellcolor[HTML]{eeedfa}-1413 &\cellcolor[HTML]{d2cff2}-3795 &\cellcolor[HTML]{9992e2}-8746 \\
hin\_Deva &0 &0 &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-40 &\cellcolor[HTML]{fcfcfe}-239 &\cellcolor[HTML]{f7f6fc}-655 & &0 &0 &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-25 &\cellcolor[HTML]{fcfcfe}-218 &\cellcolor[HTML]{f7f7fc}-632 \\
hrv\_Latn &\cellcolor[HTML]{fefefe}-65 &\cellcolor[HTML]{fcfcfe}-224 &\cellcolor[HTML]{f7f6fc}-656 &\cellcolor[HTML]{ebe9f9}-1691 &\cellcolor[HTML]{cecbf1}-4147 &\cellcolor[HTML]{8f88e0}-9578 & &\cellcolor[HTML]{fefefe}-71 &\cellcolor[HTML]{fcfcfe}-235 &\cellcolor[HTML]{f6f6fc}-694 &\cellcolor[HTML]{eae8f9}-1780 &\cellcolor[HTML]{cdc9f1}-4277 &\cellcolor[HTML]{8d85df}-9769 \\
hun\_Latn &\cellcolor[HTML]{fefefe}-75 &\cellcolor[HTML]{fcfcfe}-174 &\cellcolor[HTML]{faf9fd}-411 &\cellcolor[HTML]{f2f2fb}-1043 &\cellcolor[HTML]{e4e2f7}-2287 &\cellcolor[HTML]{c3bfee}-5145 & &\cellcolor[HTML]{fefefe}-73 &\cellcolor[HTML]{fdfcfe}-167 &\cellcolor[HTML]{faf9fd}-404 &\cellcolor[HTML]{f3f2fb}-1018 &\cellcolor[HTML]{e4e2f7}-2266 &\cellcolor[HTML]{c4c0ee}-5054 \\
hye\_Armn &0 &0 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-22 & &\cellcolor[HTML]{fefefe}-5 &\cellcolor[HTML]{fefefe}-10 &\cellcolor[HTML]{fefefe}-21 &\cellcolor[HTML]{fefefe}-68 &\cellcolor[HTML]{fdfdfe}-143 &\cellcolor[HTML]{fbfbfe}-259 \\
ind\_Latn &\cellcolor[HTML]{fefefe}-60 &\cellcolor[HTML]{fdfdfe}-148 &\cellcolor[HTML]{fafafd}-361 &\cellcolor[HTML]{f5f5fc}-782 &\cellcolor[HTML]{eceaf9}-1629 &\cellcolor[HTML]{d9d6f4}-3248 & &\cellcolor[HTML]{fefefe}-59 &\cellcolor[HTML]{fdfdfe}-144 &\cellcolor[HTML]{fafafd}-356 &\cellcolor[HTML]{f6f5fc}-768 &\cellcolor[HTML]{ecebf9}-1608 &\cellcolor[HTML]{d9d6f4}-3232 \\
isl\_Latn &\cellcolor[HTML]{fefdfe}-86 &\cellcolor[HTML]{fdfdfe}-160 &\cellcolor[HTML]{faf9fd}-427 &\cellcolor[HTML]{f4f3fc}-923 &\cellcolor[HTML]{e8e6f8}-1983 &\cellcolor[HTML]{ccc8f0}-4369 & &\cellcolor[HTML]{fefdfe}-83 &\cellcolor[HTML]{fdfdfe}-157 &\cellcolor[HTML]{faf9fd}-418 &\cellcolor[HTML]{f4f3fc}-911 &\cellcolor[HTML]{e8e6f8}-1960 &\cellcolor[HTML]{cdc9f1}-4286 \\
ita\_Latn &\cellcolor[HTML]{fefefe}-41 &\cellcolor[HTML]{fdfdfe}-145 &\cellcolor[HTML]{fafafd}-376 &\cellcolor[HTML]{f2f1fb}-1051 &\cellcolor[HTML]{e0def6}-2596 &\cellcolor[HTML]{b5b0ea}-6332 & &\cellcolor[HTML]{fefefe}-39 &\cellcolor[HTML]{fdfdfe}-144 &\cellcolor[HTML]{fafafd}-374 &\cellcolor[HTML]{f2f2fb}-1046 &\cellcolor[HTML]{e0def6}-2600 &\cellcolor[HTML]{b5b0ea}-6314 \\
jpn\_Jpan &\cellcolor[HTML]{fefefe}-14 &\cellcolor[HTML]{fefefe}-23 &\cellcolor[HTML]{fefefe}-37 &\cellcolor[HTML]{fdfdfe}-89 &\cellcolor[HTML]{fcfcfe}-232 &\cellcolor[HTML]{f9f9fd}-479 & &\cellcolor[HTML]{fefefe}-4 &\cellcolor[HTML]{fefefe}-10 &\cellcolor[HTML]{fefefe}-36 &\cellcolor[HTML]{fefdfe}-86 &\cellcolor[HTML]{fcfcfe}-223 &\cellcolor[HTML]{f8f8fd}-552 \\
kan\_Knda &0 &0 &\cellcolor[HTML]{fefefe}-5 &\cellcolor[HTML]{fefefe}-13 &\cellcolor[HTML]{fefefe}-57 &\cellcolor[HTML]{fbfbfe}-275 & &0 &0 &\cellcolor[HTML]{fefefe}-5 &\cellcolor[HTML]{fefefe}-13 &\cellcolor[HTML]{fefefe}-58 &\cellcolor[HTML]{fbfbfe}-281 \\
kat\_Geor &0 &\cellcolor[HTML]{fefefe}-8 &\cellcolor[HTML]{fefefe}-12 &\cellcolor[HTML]{fefefe}-17 &\cellcolor[HTML]{fefefe}-37 &\cellcolor[HTML]{fdfdfe}-104 & &\cellcolor[HTML]{fefefe}-5 &\cellcolor[HTML]{fefefe}-32 &\cellcolor[HTML]{fefefe}-75 &\cellcolor[HTML]{fcfcfe}-176 &\cellcolor[HTML]{faf9fd}-414 &\cellcolor[HTML]{f5f4fc}-828 \\
kaz\_Cyrl &\cellcolor[HTML]{fefefe}-74 &\cellcolor[HTML]{fcfcfe}-198 &\cellcolor[HTML]{faf9fd}-409 &\cellcolor[HTML]{f4f3fc}-902 &\cellcolor[HTML]{e8e6f8}-1971 &\cellcolor[HTML]{d1cef2}-3939 & &\cellcolor[HTML]{fefefe}-61 &\cellcolor[HTML]{fcfcfe}-182 &\cellcolor[HTML]{fafafd}-367 &\cellcolor[HTML]{f5f4fc}-855 &\cellcolor[HTML]{eae8f9}-1810 &\cellcolor[HTML]{d3cff2}-3790 \\
khk\_Cyrl &\cellcolor[HTML]{fefefe}-68 &\cellcolor[HTML]{fdfdfe}-152 &\cellcolor[HTML]{fafafd}-389 &\cellcolor[HTML]{f4f3fc}-914 &\cellcolor[HTML]{e9e8f9}-1833 &\cellcolor[HTML]{d5d2f3}-3621 & &\cellcolor[HTML]{fefefe}-67 &\cellcolor[HTML]{fdfdfe}-145 &\cellcolor[HTML]{fafafd}-358 &\cellcolor[HTML]{f5f4fc}-806 &\cellcolor[HTML]{ebeaf9}-1644 &\cellcolor[HTML]{dad8f5}-3116 \\
khm\_Khmr &0 &0 &0 &\cellcolor[HTML]{fefefe}-3 &\cellcolor[HTML]{fefefe}-18 &\cellcolor[HTML]{fdfdfe}-119 & &0 &0 &0 &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-23 &\cellcolor[HTML]{fdfdfe}-136 \\
kir\_Cyrl &\cellcolor[HTML]{fefefe}-68 &\cellcolor[HTML]{fdfdfe}-155 &\cellcolor[HTML]{fafafd}-402 &\cellcolor[HTML]{f4f3fb}-938 &\cellcolor[HTML]{e7e6f8}-2005 &\cellcolor[HTML]{cdcaf1}-4238 & &\cellcolor[HTML]{fefefe}-60 &\cellcolor[HTML]{fdfdfe}-135 &\cellcolor[HTML]{fafafd}-361 &\cellcolor[HTML]{f5f4fc}-825 &\cellcolor[HTML]{eae8f9}-1774 &\cellcolor[HTML]{d2cff2}-3810 \\
kor\_Hang &\cellcolor[HTML]{fefefe}-13 &\cellcolor[HTML]{fefefe}-21 &\cellcolor[HTML]{fefefe}-36 &\cellcolor[HTML]{fefefe}-77 &\cellcolor[HTML]{fdfdfe}-146 &\cellcolor[HTML]{fbfbfe}-307 & &\cellcolor[HTML]{fefefe}-5 &\cellcolor[HTML]{fefefe}-15 &\cellcolor[HTML]{fefefe}-32 &\cellcolor[HTML]{fefefe}-71 &\cellcolor[HTML]{fdfdfe}-142 &\cellcolor[HTML]{fbfbfe}-304 \\
lit\_Latn &\cellcolor[HTML]{fefefe}-71 &\cellcolor[HTML]{fcfcfe}-177 &\cellcolor[HTML]{f9f9fd}-455 &\cellcolor[HTML]{f2f1fb}-1118 &\cellcolor[HTML]{e2e0f7}-2456 &\cellcolor[HTML]{c1bded}-5307 & &\cellcolor[HTML]{fefefe}-64 &\cellcolor[HTML]{fcfcfe}-178 &\cellcolor[HTML]{f9f9fd}-450 &\cellcolor[HTML]{f2f1fb}-1105 &\cellcolor[HTML]{e2e0f7}-2447 &\cellcolor[HTML]{c2bdee}-5255 \\
lvs\_Latn &\cellcolor[HTML]{fefefe}-77 &\cellcolor[HTML]{fcfcfe}-188 &\cellcolor[HTML]{fbfafd}-338 &\cellcolor[HTML]{f5f5fc}-786 &\cellcolor[HTML]{e9e8f9}-1817 &\cellcolor[HTML]{d2cef2}-3869 & &\cellcolor[HTML]{fefefe}-66 &\cellcolor[HTML]{fcfcfe}-177 &\cellcolor[HTML]{fbfafd}-328 &\cellcolor[HTML]{f5f5fc}-783 &\cellcolor[HTML]{eae8f9}-1796 &\cellcolor[HTML]{d2cff2}-3800 \\
mal\_Mlym &0 &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-5 &\cellcolor[HTML]{fefefe}-13 &\cellcolor[HTML]{fefefe}-74 &\cellcolor[HTML]{fafafd}-387 & &0 &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-5 &\cellcolor[HTML]{fefefe}-15 &\cellcolor[HTML]{fefefe}-71 &\cellcolor[HTML]{fafafd}-384 \\
mar\_Deva &\cellcolor[HTML]{fefefe}-3 &\cellcolor[HTML]{fefefe}-14 &\cellcolor[HTML]{fefefe}-17 &\cellcolor[HTML]{fefefe}-41 &\cellcolor[HTML]{fdfcfe}-163 &\cellcolor[HTML]{f8f8fd}-521 & &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-5 &\cellcolor[HTML]{fefefe}-18 &\cellcolor[HTML]{fdfdfe}-130 &\cellcolor[HTML]{f9f8fd}-501 \\
mkd\_Cyrl &\cellcolor[HTML]{fdfdfe}-91 &\cellcolor[HTML]{fbfbfe}-263 &\cellcolor[HTML]{f7f7fc}-627 &\cellcolor[HTML]{ecebf9}-1562 &\cellcolor[HTML]{d4d1f3}-3647 &\cellcolor[HTML]{9e98e4}-8282 & &\cellcolor[HTML]{fdfdfe}-103 &\cellcolor[HTML]{fbfbfd}-319 &\cellcolor[HTML]{f6f5fc}-767 &\cellcolor[HTML]{e9e8f9}-1830 &\cellcolor[HTML]{cecaf1}-4215 &\cellcolor[HTML]{948ce1}-9192 \\
mya\_Mymr &\cellcolor[HTML]{fefefe}-16 &\cellcolor[HTML]{fefefe}-20 &\cellcolor[HTML]{fefefe}-43 &\cellcolor[HTML]{fefdfe}-85 &\cellcolor[HTML]{fcfcfe}-200 &\cellcolor[HTML]{f8f8fd}-518 & &\cellcolor[HTML]{fefefe}-15 &\cellcolor[HTML]{fefefe}-21 &\cellcolor[HTML]{fefefe}-46 &\cellcolor[HTML]{fdfdfe}-90 &\cellcolor[HTML]{fcfcfe}-213 &\cellcolor[HTML]{f8f8fd}-549 \\
nld\_Latn &\cellcolor[HTML]{fefefe}-67 &\cellcolor[HTML]{fdfcfe}-168 &\cellcolor[HTML]{fafafd}-388 &\cellcolor[HTML]{f4f4fc}-866 &\cellcolor[HTML]{eae9f9}-1757 &\cellcolor[HTML]{d3d0f2}-3742 & &\cellcolor[HTML]{fefefe}-65 &\cellcolor[HTML]{fdfcfe}-163 &\cellcolor[HTML]{fafafd}-384 &\cellcolor[HTML]{f4f4fc}-863 &\cellcolor[HTML]{eae9f9}-1751 &\cellcolor[HTML]{d3d0f3}-3736 \\
nno\_Latn &\cellcolor[HTML]{fefefe}-46 &\cellcolor[HTML]{fdfdfe}-108 &\cellcolor[HTML]{fcfbfe}-253 &\cellcolor[HTML]{f8f8fd}-552 &\cellcolor[HTML]{efeefa}-1303 &\cellcolor[HTML]{dbd8f5}-3095 & &\cellcolor[HTML]{fefefe}-48 &\cellcolor[HTML]{fdfdfe}-109 &\cellcolor[HTML]{fcfbfe}-249 &\cellcolor[HTML]{f8f7fd}-564 &\cellcolor[HTML]{efeefa}-1317 &\cellcolor[HTML]{dbd8f5}-3072 \\
nob\_Latn &\cellcolor[HTML]{fefefe}-36 &\cellcolor[HTML]{fdfdfe}-101 &\cellcolor[HTML]{fcfbfe}-255 &\cellcolor[HTML]{f8f8fd}-530 &\cellcolor[HTML]{f0effb}-1208 &\cellcolor[HTML]{e0def6}-2639 & &\cellcolor[HTML]{fefefe}-38 &\cellcolor[HTML]{fdfdfe}-101 &\cellcolor[HTML]{fcfbfe}-255 &\cellcolor[HTML]{f8f8fd}-530 &\cellcolor[HTML]{f0effb}-1211 &\cellcolor[HTML]{e0def6}-2626 \\
npi\_Deva &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-3 &\cellcolor[HTML]{fefefe}-6 &\cellcolor[HTML]{fefefe}-18 &\cellcolor[HTML]{fefefe}-49 &\cellcolor[HTML]{fdfcfe}-169 & &0 &0 &0 &\cellcolor[HTML]{fefefe}-3 &\cellcolor[HTML]{fefefe}-20 &\cellcolor[HTML]{fdfdfe}-131 \\
ory\_Orya &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-3 &\cellcolor[HTML]{fefefe}-3 &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-46 &\cellcolor[HTML]{fbfbfe}-296 & &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-3 &\cellcolor[HTML]{fefefe}-3 &\cellcolor[HTML]{fefefe}-4 &\cellcolor[HTML]{fefefe}-74 &\cellcolor[HTML]{fbfafd}-324 \\
pan\_Guru &0 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-4 &\cellcolor[HTML]{fefefe}-18 &\cellcolor[HTML]{fdfdfe}-91 &\cellcolor[HTML]{fbfafd}-340 & &0 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-4 &\cellcolor[HTML]{fefefe}-18 &\cellcolor[HTML]{fdfdfe}-89 &\cellcolor[HTML]{fafafd}-354 \\
pol\_Latn &\cellcolor[HTML]{fefefe}-56 &\cellcolor[HTML]{fcfcfe}-192 &\cellcolor[HTML]{f8f7fd}-567 &\cellcolor[HTML]{eeedfa}-1408 &\cellcolor[HTML]{d7d4f3}-3443 &\cellcolor[HTML]{a39ce5}-7898 & &\cellcolor[HTML]{fefefe}-56 &\cellcolor[HTML]{fcfcfe}-195 &\cellcolor[HTML]{f8f8fd}-545 &\cellcolor[HTML]{efeefa}-1363 &\cellcolor[HTML]{d8d5f4}-3363 &\cellcolor[HTML]{a49ee5}-7806 \\
por\_Latn &\cellcolor[HTML]{fefefe}-19 &\cellcolor[HTML]{fefefe}-52 &\cellcolor[HTML]{fdfdfe}-158 &\cellcolor[HTML]{faf9fd}-419 &\cellcolor[HTML]{f3f2fb}-998 &\cellcolor[HTML]{e5e4f8}-2165 & &\cellcolor[HTML]{fefefe}-21 &\cellcolor[HTML]{fefefe}-54 &\cellcolor[HTML]{fdfcfe}-166 &\cellcolor[HTML]{faf9fd}-428 &\cellcolor[HTML]{f3f2fb}-1003 &\cellcolor[HTML]{e5e4f8}-2162 \\
ron\_Latn &\cellcolor[HTML]{fefefe}-50 &\cellcolor[HTML]{fdfdfe}-138 &\cellcolor[HTML]{fbfafd}-337 &\cellcolor[HTML]{f5f4fc}-817 &\cellcolor[HTML]{e8e6f8}-1975 &\cellcolor[HTML]{cac7f0}-4487 & &\cellcolor[HTML]{fefefe}-48 &\cellcolor[HTML]{fdfdfe}-140 &\cellcolor[HTML]{fbfafd}-338 &\cellcolor[HTML]{f5f4fc}-815 &\cellcolor[HTML]{e8e6f8}-1981 &\cellcolor[HTML]{cac7f0}-4499 \\
rus\_Cyrl &\cellcolor[HTML]{fefdfe}-81 &\cellcolor[HTML]{fdfdfe}-134 &\cellcolor[HTML]{fcfbfe}-244 &\cellcolor[HTML]{f9f9fd}-462 &\cellcolor[HTML]{f4f3fc}-922 &\cellcolor[HTML]{e8e7f8}-1898 & &\cellcolor[HTML]{fefefe}-22 &\cellcolor[HTML]{fefefe}-59 &\cellcolor[HTML]{fdfdfe}-139 &\cellcolor[HTML]{fbfafd}-328 &\cellcolor[HTML]{f5f5fc}-791 &\cellcolor[HTML]{ebe9f9}-1699 \\
sin\_Sinh &\cellcolor[HTML]{fefefe}-58 &\cellcolor[HTML]{fdfdfe}-110 &\cellcolor[HTML]{fcfcfe}-206 &\cellcolor[HTML]{faf9fd}-409 &\cellcolor[HTML]{f5f5fc}-783 &\cellcolor[HTML]{edecfa}-1494 & &\cellcolor[HTML]{fefefe}-60 &\cellcolor[HTML]{fdfdfe}-110 &\cellcolor[HTML]{fcfcfe}-207 &\cellcolor[HTML]{faf9fd}-410 &\cellcolor[HTML]{f5f5fc}-783 &\cellcolor[HTML]{edecfa}-1494 \\
slk\_Latn &\cellcolor[HTML]{fdfdfe}-103 &\cellcolor[HTML]{fcfcfe}-197 &\cellcolor[HTML]{f9f9fd}-443 &\cellcolor[HTML]{f3f2fb}-1019 &\cellcolor[HTML]{e3e1f7}-2367 &\cellcolor[HTML]{c0bced}-5391 & &\cellcolor[HTML]{fefefe}-52 &\cellcolor[HTML]{fdfdfe}-152 &\cellcolor[HTML]{faf9fd}-408 &\cellcolor[HTML]{f3f2fb}-990 &\cellcolor[HTML]{e3e1f7}-2395 &\cellcolor[HTML]{bfbaed}-5512 \\
slv\_Latn &\cellcolor[HTML]{fdfdfe}-91 &\cellcolor[HTML]{fcfcfe}-229 &\cellcolor[HTML]{f8f7fd}-599 &\cellcolor[HTML]{edecfa}-1491 &\cellcolor[HTML]{d6d3f3}-3524 &\cellcolor[HTML]{a39ce5}-7898 & &\cellcolor[HTML]{fefefe}-73 &\cellcolor[HTML]{fcfcfe}-210 &\cellcolor[HTML]{f8f7fd}-595 &\cellcolor[HTML]{edecfa}-1479 &\cellcolor[HTML]{d5d2f3}-3545 &\cellcolor[HTML]{a39de5}-7863 \\
spa\_Latn &\cellcolor[HTML]{fefefe}-17 &\cellcolor[HTML]{fefefe}-44 &\cellcolor[HTML]{fdfcfe}-170 &\cellcolor[HTML]{f9f9fd}-475 &\cellcolor[HTML]{f0effb}-1224 &\cellcolor[HTML]{dbd9f5}-3033 & &\cellcolor[HTML]{fefefe}-18 &\cellcolor[HTML]{fefefe}-44 &\cellcolor[HTML]{fdfcfe}-165 &\cellcolor[HTML]{f9f9fd}-473 &\cellcolor[HTML]{f0effb}-1218 &\cellcolor[HTML]{dbd9f5}-3030 \\
srp\_Cyrl &\cellcolor[HTML]{fefefe}-45 &\cellcolor[HTML]{fdfdfe}-119 &\cellcolor[HTML]{fbfbfe}-284 &\cellcolor[HTML]{f7f7fc}-625 &\cellcolor[HTML]{edecfa}-1525 &\cellcolor[HTML]{d8d5f4}-3351 & &\cellcolor[HTML]{fefefe}-23 &\cellcolor[HTML]{fdfdfe}-90 &\cellcolor[HTML]{fbfbfe}-267 &\cellcolor[HTML]{f8f7fd}-595 &\cellcolor[HTML]{edecfa}-1478 &\cellcolor[HTML]{d9d6f4}-3248 \\
swe\_Latn &\cellcolor[HTML]{fefefe}-68 &\cellcolor[HTML]{fdfdfe}-156 &\cellcolor[HTML]{fafafd}-350 &\cellcolor[HTML]{f6f5fc}-751 &\cellcolor[HTML]{eeecfa}-1459 &\cellcolor[HTML]{dbd8f5}-3064 & &\cellcolor[HTML]{fefefe}-68 &\cellcolor[HTML]{fdfdfe}-154 &\cellcolor[HTML]{fbfafd}-341 &\cellcolor[HTML]{f6f5fc}-737 &\cellcolor[HTML]{eeedfa}-1437 &\cellcolor[HTML]{dbd9f5}-3052 \\
swh\_Latn &\cellcolor[HTML]{fdfdfe}-157 &\cellcolor[HTML]{f9f9fd}-443 &\cellcolor[HTML]{f1f0fb}-1146 &\cellcolor[HTML]{e1dff6}-2513 &\cellcolor[HTML]{beb9ed}-5583 &\cellcolor[HTML]{7369d8}-12025 & &\cellcolor[HTML]{fdfcfe}-163 &\cellcolor[HTML]{f9f9fd}-468 &\cellcolor[HTML]{f1f0fb}-1185 &\cellcolor[HTML]{e1def6}-2581 &\cellcolor[HTML]{bdb8ec}-5670 &\cellcolor[HTML]{7268d8}-12159 \\
tam\_Taml &0 &0 &\cellcolor[HTML]{fefefe}-3 &\cellcolor[HTML]{fefefe}-34 &\cellcolor[HTML]{fdfdfe}-156 &\cellcolor[HTML]{f8f7fd}-570 & &0 &0 &\cellcolor[HTML]{fefefe}-3 &\cellcolor[HTML]{fefefe}-37 &\cellcolor[HTML]{fdfdfe}-151 &\cellcolor[HTML]{f7f7fd}-607 \\
tel\_Telu &0 &0 &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-13 &\cellcolor[HTML]{fefdfe}-81 &\cellcolor[HTML]{faf9fd}-407 & &0 &0 &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-13 &\cellcolor[HTML]{fefefe}-76 &\cellcolor[HTML]{faf9fd}-405 \\
tha\_Thai &\cellcolor[HTML]{fefefe}-9 &\cellcolor[HTML]{fefefe}-15 &\cellcolor[HTML]{fefefe}-28 &\cellcolor[HTML]{fefefe}-54 &\cellcolor[HTML]{fdfdfe}-105 &\cellcolor[HTML]{fcfbfe}-250 & &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-7 &\cellcolor[HTML]{fefefe}-34 &\cellcolor[HTML]{fefefe}-70 &\cellcolor[HTML]{fdfdfe}-136 &\cellcolor[HTML]{fbfafd}-334 \\
tur\_Latn &\cellcolor[HTML]{fcfcfe}-232 &\cellcolor[HTML]{fafafd}-359 &\cellcolor[HTML]{f7f6fc}-645 &\cellcolor[HTML]{f0effb}-1245 &\cellcolor[HTML]{e0def6}-2588 &\cellcolor[HTML]{c1bdee}-5275 & &\cellcolor[HTML]{fefefe}-51 &\cellcolor[HTML]{fdfdfe}-141 &\cellcolor[HTML]{fafafd}-358 &\cellcolor[HTML]{f5f4fc}-856 &\cellcolor[HTML]{e8e6f8}-1974 &\cellcolor[HTML]{ccc8f0}-4377 \\
ukr\_Cyrl &\cellcolor[HTML]{fefefe}-76 &\cellcolor[HTML]{fdfdfe}-123 &\cellcolor[HTML]{fcfcfe}-225 &\cellcolor[HTML]{f9f9fd}-473 &\cellcolor[HTML]{f3f2fb}-985 &\cellcolor[HTML]{e7e5f8}-2063 & &\cellcolor[HTML]{fefefe}-28 &\cellcolor[HTML]{fefefe}-75 &\cellcolor[HTML]{fcfcfe}-192 &\cellcolor[HTML]{f9f9fd}-470 &\cellcolor[HTML]{f3f2fb}-1008 &\cellcolor[HTML]{e5e3f7}-2206 \\
urd\_Arab &\cellcolor[HTML]{fefefe}-48 &\cellcolor[HTML]{fdfdfe}-98 &\cellcolor[HTML]{fcfcfe}-192 &\cellcolor[HTML]{faf9fd}-420 &\cellcolor[HTML]{f5f5fc}-783 &\cellcolor[HTML]{eeedfa}-1393 & &\cellcolor[HTML]{fefefe}-53 &\cellcolor[HTML]{fdfdfe}-121 &\cellcolor[HTML]{fbfbfe}-310 &\cellcolor[HTML]{f8f7fd}-596 &\cellcolor[HTML]{f3f2fb}-1012 &\cellcolor[HTML]{ebeaf9}-1664 \\
uzn\_Latn &\cellcolor[HTML]{fdfdfe}-105 &\cellcolor[HTML]{fbfbfe}-297 &\cellcolor[HTML]{f6f5fc}-733 &\cellcolor[HTML]{ebeaf9}-1648 &\cellcolor[HTML]{d6d3f3}-3510 &\cellcolor[HTML]{a7a0e6}-7571 & &\cellcolor[HTML]{fdfdfe}-105 &\cellcolor[HTML]{fbfbfe}-288 &\cellcolor[HTML]{f6f5fc}-736 &\cellcolor[HTML]{ebeaf9}-1677 &\cellcolor[HTML]{d5d2f3}-3577 &\cellcolor[HTML]{a59fe6}-7690 \\
vie\_Latn &\cellcolor[HTML]{fefefe}-35 &\cellcolor[HTML]{fefefe}-48 &\cellcolor[HTML]{fefefe}-65 &\cellcolor[HTML]{fdfdfe}-148 &\cellcolor[HTML]{fafafd}-375 &\cellcolor[HTML]{f3f2fb}-1010 & &\cellcolor[HTML]{fefefe}-6 &\cellcolor[HTML]{fefefe}-10 &\cellcolor[HTML]{fefefe}-26 &\cellcolor[HTML]{fdfdfe}-121 &\cellcolor[HTML]{fbfafd}-343 &\cellcolor[HTML]{f3f2fb}-982 \\
zsm\_Latn &\cellcolor[HTML]{fefefe}-66 &\cellcolor[HTML]{fdfdfe}-145 &\cellcolor[HTML]{fafafd}-369 &\cellcolor[HTML]{f5f4fc}-823 &\cellcolor[HTML]{ebeaf9}-1675 &\cellcolor[HTML]{dad7f4}-3153 & &\cellcolor[HTML]{fefefe}-65 &\cellcolor[HTML]{fdfdfe}-141 &\cellcolor[HTML]{fafafd}-360 &\cellcolor[HTML]{f5f5fc}-802 &\cellcolor[HTML]{ebeaf9}-1641 &\cellcolor[HTML]{dad8f4}-3119 \\
\bottomrule
\end{tabular}

\begin{tabular}{lcccccccccccccc}\toprule
&\multicolumn{6}{c}{\textbf{Mistral Nemo extension}} & &\multicolumn{6}{c}{\textbf{Llama-2 extension}} \\\cmidrule{2-7}\cmidrule{9-14}
\textbf{Lang.} &\textbf{1000} &\textbf{2000} &\textbf{4000} &\textbf{8000} &\textbf{16000} &\textbf{32000} & &\textbf{1000} &\textbf{2000} &\textbf{4000} &\textbf{8000} &\textbf{16000} &\textbf{32000} \\\midrule
afr\_Latn &\cellcolor[HTML]{fefefe}-62 &\cellcolor[HTML]{fdfdfe}-132 &\cellcolor[HTML]{fbfafd}-329 &\cellcolor[HTML]{f5f5fc}-793 &\cellcolor[HTML]{eceaf9}-1634 &\cellcolor[HTML]{d8d6f4}-3290 & &\cellcolor[HTML]{fefefe}-59 &\cellcolor[HTML]{fdfdfe}-149 &\cellcolor[HTML]{fbfafd}-330 &\cellcolor[HTML]{f6f5fc}-753 &\cellcolor[HTML]{eceaf9}-1617 &\cellcolor[HTML]{dad7f4}-3187 \\
als\_Latn &\cellcolor[HTML]{fefdfe}-83 &\cellcolor[HTML]{fcfcfe}-197 &\cellcolor[HTML]{f9f9fd}-437 &\cellcolor[HTML]{f4f4fc}-880 &\cellcolor[HTML]{e9e8f9}-1833 &\cellcolor[HTML]{d4d1f3}-3661 & &\cellcolor[HTML]{fefefe}-78 &\cellcolor[HTML]{fcfcfe}-226 &\cellcolor[HTML]{f9f8fd}-514 &\cellcolor[HTML]{f2f1fb}-1070 &\cellcolor[HTML]{e6e5f8}-2074 &\cellcolor[HTML]{cecaf1}-4218 \\
arb\_Arab &\cellcolor[HTML]{fefefe}-9 &\cellcolor[HTML]{fefefe}-28 &\cellcolor[HTML]{fefefe}-58 &\cellcolor[HTML]{fdfdfe}-114 &\cellcolor[HTML]{fcfbfe}-245 &\cellcolor[HTML]{f8f7fd}-577 & &0 &0 &0 &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-4 &\cellcolor[HTML]{fefefe}-12 \\
ary\_Arab &\cellcolor[HTML]{fefefe}-10 &\cellcolor[HTML]{fefefe}-24 &\cellcolor[HTML]{fefefe}-55 &\cellcolor[HTML]{fdfdfe}-103 &\cellcolor[HTML]{fbfbfe}-264 &\cellcolor[HTML]{f8f8fd}-550 & &0 &0 &0 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-6 \\
azj\_Latn &\cellcolor[HTML]{fdfdfe}-104 &\cellcolor[HTML]{fcfcfe}-225 &\cellcolor[HTML]{f8f8fd}-552 &\cellcolor[HTML]{f1f0fb}-1167 &\cellcolor[HTML]{e1def6}-2577 &\cellcolor[HTML]{c1bced}-5327 & &\cellcolor[HTML]{fdfdfe}-128 &\cellcolor[HTML]{fcfbfe}-255 &\cellcolor[HTML]{f7f7fc}-631 &\cellcolor[HTML]{edecfa}-1474 &\cellcolor[HTML]{d9d6f4}-3261 &\cellcolor[HTML]{b2ace9}-6616 \\
bel\_Cyrl &\cellcolor[HTML]{fefefe}-52 &\cellcolor[HTML]{fdfdfe}-158 &\cellcolor[HTML]{faf9fd}-408 &\cellcolor[HTML]{f3f3fb}-950 &\cellcolor[HTML]{e5e3f8}-2178 &\cellcolor[HTML]{c7c3ef}-4767 & &\cellcolor[HTML]{fefefe}-22 &\cellcolor[HTML]{fefefe}-65 &\cellcolor[HTML]{fdfdfe}-138 &\cellcolor[HTML]{fbfbfd}-319 &\cellcolor[HTML]{f5f5fc}-779 &\cellcolor[HTML]{ebe9f9}-1708 \\
ben\_Beng &\cellcolor[HTML]{fefefe}-6 &\cellcolor[HTML]{fefefe}-19 &\cellcolor[HTML]{fefefe}-45 &\cellcolor[HTML]{fdfdfe}-93 &\cellcolor[HTML]{fdfcfe}-167 &\cellcolor[HTML]{fafafd}-358 & &0 &0 &0 &0 &0 &\cellcolor[HTML]{fefefe}-1 \\
bos\_Latn &\cellcolor[HTML]{fefefe}-24 &\cellcolor[HTML]{fefefe}-79 &\cellcolor[HTML]{fbfbfe}-292 &\cellcolor[HTML]{f4f3fc}-908 &\cellcolor[HTML]{e2e0f7}-2462 &\cellcolor[HTML]{b6b1ea}-6251 & &\cellcolor[HTML]{fefefe}-56 &\cellcolor[HTML]{fcfcfe}-197 &\cellcolor[HTML]{f8f8fd}-545 &\cellcolor[HTML]{edecfa}-1524 &\cellcolor[HTML]{d4d1f3}-3658 &\cellcolor[HTML]{9c95e3}-8530 \\
bul\_Cyrl &\cellcolor[HTML]{fefefe}-21 &\cellcolor[HTML]{fefefe}-60 &\cellcolor[HTML]{fdfdfe}-161 &\cellcolor[HTML]{faf9fd}-426 &\cellcolor[HTML]{f2f1fb}-1081 &\cellcolor[HTML]{e0def6}-2623 & &\cellcolor[HTML]{fefefe}-16 &\cellcolor[HTML]{fefefe}-47 &\cellcolor[HTML]{fdfdfe}-125 &\cellcolor[HTML]{fbfbfe}-300 &\cellcolor[HTML]{f7f6fc}-673 &\cellcolor[HTML]{edecfa}-1510 \\
cat\_Latn &\cellcolor[HTML]{fefefe}-33 &\cellcolor[HTML]{fefdfe}-84 &\cellcolor[HTML]{fcfcfe}-204 &\cellcolor[HTML]{f9f9fd}-480 &\cellcolor[HTML]{f2f1fb}-1058 &\cellcolor[HTML]{e2e0f7}-2439 & &\cellcolor[HTML]{fefefe}-37 &\cellcolor[HTML]{fdfdfe}-115 &\cellcolor[HTML]{fbfbfe}-285 &\cellcolor[HTML]{f7f6fc}-648 &\cellcolor[HTML]{eeedfa}-1442 &\cellcolor[HTML]{dbd8f5}-3082 \\
ces\_Latn &\cellcolor[HTML]{fefefe}-31 &\cellcolor[HTML]{fefdfe}-85 &\cellcolor[HTML]{fcfbfe}-254 &\cellcolor[HTML]{f6f5fc}-745 &\cellcolor[HTML]{eae8f9}-1809 &\cellcolor[HTML]{cecaf1}-4197 & &\cellcolor[HTML]{fefefe}-52 &\cellcolor[HTML]{fdfdfe}-150 &\cellcolor[HTML]{fafafd}-377 &\cellcolor[HTML]{f4f3fc}-920 &\cellcolor[HTML]{e5e3f7}-2199 &\cellcolor[HTML]{c6c2ef}-4876 \\
cmn\_Hani &0 &0 &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-9 &\cellcolor[HTML]{fefefe}-32 &\cellcolor[HTML]{fefefe}-80 & &0 &0 &0 &0 &0 &\cellcolor[HTML]{fefefe}-7 \\
dan\_Latn &\cellcolor[HTML]{fefefe}-53 &\cellcolor[HTML]{fdfdfe}-123 &\cellcolor[HTML]{fbfbfe}-285 &\cellcolor[HTML]{f7f7fc}-633 &\cellcolor[HTML]{efeefa}-1360 &\cellcolor[HTML]{dcdaf5}-2964 & &\cellcolor[HTML]{fefefe}-56 &\cellcolor[HTML]{fdfdfe}-132 &\cellcolor[HTML]{fbfbfe}-304 &\cellcolor[HTML]{f6f6fc}-694 &\cellcolor[HTML]{eeedfa}-1424 &\cellcolor[HTML]{dcd9f5}-3007 \\
deu\_Latn &\cellcolor[HTML]{fefefe}-15 &\cellcolor[HTML]{fefefe}-43 &\cellcolor[HTML]{fdfdfe}-136 &\cellcolor[HTML]{fafafd}-376 &\cellcolor[HTML]{f4f3fc}-922 &\cellcolor[HTML]{e5e3f7}-2237 & &\cellcolor[HTML]{fefefe}-21 &\cellcolor[HTML]{fefefe}-73 &\cellcolor[HTML]{fcfcfe}-209 &\cellcolor[HTML]{f8f8fd}-532 &\cellcolor[HTML]{f1f0fb}-1155 &\cellcolor[HTML]{e1dff6}-2543 \\
ekk\_Latn &\cellcolor[HTML]{fefefe}-73 &\cellcolor[HTML]{fdfdfe}-160 &\cellcolor[HTML]{fafafd}-367 &\cellcolor[HTML]{f4f4fc}-863 &\cellcolor[HTML]{e8e6f8}-1958 &\cellcolor[HTML]{cdc9f1}-4310 & &\cellcolor[HTML]{fefefe}-66 &\cellcolor[HTML]{fcfcfe}-176 &\cellcolor[HTML]{fafafd}-382 &\cellcolor[HTML]{f4f3fc}-911 &\cellcolor[HTML]{e7e5f8}-2044 &\cellcolor[HTML]{cac6f0}-4528 \\
ell\_Grek &\cellcolor[HTML]{fefefe}-23 &\cellcolor[HTML]{fefefe}-58 &\cellcolor[HTML]{fcfcfe}-177 &\cellcolor[HTML]{faf9fd}-424 &\cellcolor[HTML]{f3f2fb}-1023 &\cellcolor[HTML]{e4e2f7}-2265 & &0 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-6 &\cellcolor[HTML]{fefefe}-8 &\cellcolor[HTML]{fefefe}-37 \\
eus\_Latn &\cellcolor[HTML]{fefefe}-76 &\cellcolor[HTML]{fcfcfe}-206 &\cellcolor[HTML]{f8f8fd}-518 &\cellcolor[HTML]{f0effb}-1244 &\cellcolor[HTML]{dedbf5}-2835 &\cellcolor[HTML]{b6b1eb}-6223 & &\cellcolor[HTML]{fdfdfe}-113 &\cellcolor[HTML]{fbfbfd}-320 &\cellcolor[HTML]{f6f6fc}-724 &\cellcolor[HTML]{eceaf9}-1622 &\cellcolor[HTML]{d5d2f3}-3586 &\cellcolor[HTML]{a9a3e7}-7406 \\
fas\_Arab &\cellcolor[HTML]{fefefe}-16 &\cellcolor[HTML]{fefefe}-51 &\cellcolor[HTML]{fdfdfe}-144 &\cellcolor[HTML]{fafafd}-348 &\cellcolor[HTML]{f6f5fc}-725 &\cellcolor[HTML]{eeedfa}-1440 & &0 &0 &0 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-4 &\cellcolor[HTML]{fefefe}-30 \\
fin\_Latn &\cellcolor[HTML]{fefefe}-33 &\cellcolor[HTML]{fdfdfe}-104 &\cellcolor[HTML]{fbfbfe}-287 &\cellcolor[HTML]{f5f4fc}-811 &\cellcolor[HTML]{e7e5f8}-2016 &\cellcolor[HTML]{c6c2ef}-4868 & &\cellcolor[HTML]{fefefe}-63 &\cellcolor[HTML]{fdfcfe}-164 &\cellcolor[HTML]{faf9fd}-422 &\cellcolor[HTML]{f3f2fb}-1014 &\cellcolor[HTML]{e3e1f7}-2390 &\cellcolor[HTML]{bfbaed}-5509 \\
fra\_Latn &\cellcolor[HTML]{fefefe}-19 &\cellcolor[HTML]{fefefe}-46 &\cellcolor[HTML]{fdfdfe}-107 &\cellcolor[HTML]{fcfbfe}-255 &\cellcolor[HTML]{f7f7fd}-613 &\cellcolor[HTML]{ecebf9}-1578 & &\cellcolor[HTML]{fefefe}-28 &\cellcolor[HTML]{fefefe}-78 &\cellcolor[HTML]{fcfcfe}-185 &\cellcolor[HTML]{f9f9fd}-445 &\cellcolor[HTML]{f3f2fb}-1023 &\cellcolor[HTML]{e5e3f7}-2203 \\
glg\_Latn &\cellcolor[HTML]{fefefe}-29 &\cellcolor[HTML]{fefefe}-70 &\cellcolor[HTML]{fdfdfe}-156 &\cellcolor[HTML]{fbfafd}-335 &\cellcolor[HTML]{f5f5fc}-784 &\cellcolor[HTML]{e9e7f8}-1888 & &\cellcolor[HTML]{fefefe}-41 &\cellcolor[HTML]{fdfdfe}-102 &\cellcolor[HTML]{fcfcfe}-232 &\cellcolor[HTML]{f8f8fd}-551 &\cellcolor[HTML]{f1f0fb}-1179 &\cellcolor[HTML]{e0ddf6}-2659 \\
guj\_Gujr &\cellcolor[HTML]{fefefe}-20 &\cellcolor[HTML]{fefefe}-43 &\cellcolor[HTML]{fefefe}-80 &\cellcolor[HTML]{fdfcfe}-165 &\cellcolor[HTML]{fbfbfe}-301 &\cellcolor[HTML]{f7f7fd}-616 & &0 &0 &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-3 &\cellcolor[HTML]{fefefe}-10 &\cellcolor[HTML]{fefefe}-25 \\
heb\_Hebr &\cellcolor[HTML]{fefefe}-28 &\cellcolor[HTML]{fdfdfe}-115 &\cellcolor[HTML]{f9f9fd}-453 &\cellcolor[HTML]{edecfa}-1511 &\cellcolor[HTML]{cecbf1}-4173 &\cellcolor[HTML]{8d85df}-9750 & &0 &0 &0 &0 &0 &\cellcolor[HTML]{fefefe}-1 \\
hin\_Deva &\cellcolor[HTML]{fefefe}-17 &\cellcolor[HTML]{fefefe}-27 &\cellcolor[HTML]{fefefe}-55 &\cellcolor[HTML]{fdfdfe}-104 &\cellcolor[HTML]{fcfcfe}-202 &\cellcolor[HTML]{fafafd}-369 & &0 &0 &0 &0 &\cellcolor[HTML]{fefefe}-8 &\cellcolor[HTML]{fefefe}-62 \\
hrv\_Latn &\cellcolor[HTML]{fefefe}-43 &\cellcolor[HTML]{fdfdfe}-139 &\cellcolor[HTML]{faf9fd}-431 &\cellcolor[HTML]{f0effb}-1209 &\cellcolor[HTML]{dbd9f5}-3044 &\cellcolor[HTML]{a7a1e6}-7548 & &\cellcolor[HTML]{fefefe}-74 &\cellcolor[HTML]{fcfbfe}-243 &\cellcolor[HTML]{f7f6fc}-686 &\cellcolor[HTML]{ebe9f9}-1714 &\cellcolor[HTML]{cecaf1}-4212 &\cellcolor[HTML]{8f87e0}-9613 \\
hun\_Latn &\cellcolor[HTML]{fefefe}-40 &\cellcolor[HTML]{fdfdfe}-102 &\cellcolor[HTML]{fbfbfe}-292 &\cellcolor[HTML]{f7f6fc}-675 &\cellcolor[HTML]{edebfa}-1548 &\cellcolor[HTML]{d4d1f3}-3624 & &\cellcolor[HTML]{fefefe}-59 &\cellcolor[HTML]{fdfdfe}-154 &\cellcolor[HTML]{fbfafd}-333 &\cellcolor[HTML]{f5f4fc}-847 &\cellcolor[HTML]{e8e7f8}-1906 &\cellcolor[HTML]{cecbf1}-4161 \\
hye\_Armn &\cellcolor[HTML]{fefefe}-3 &\cellcolor[HTML]{fefefe}-10 &\cellcolor[HTML]{fefefe}-28 &\cellcolor[HTML]{fefefe}-54 &\cellcolor[HTML]{fdfdfe}-92 &\cellcolor[HTML]{fcfcfe}-201 & &0 &0 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-3 &\cellcolor[HTML]{fefefe}-11 \\
ind\_Latn &\cellcolor[HTML]{fefefe}-38 &\cellcolor[HTML]{fdfdfe}-105 &\cellcolor[HTML]{fcfbfe}-256 &\cellcolor[HTML]{f8f7fd}-568 &\cellcolor[HTML]{efeefa}-1339 &\cellcolor[HTML]{dedcf5}-2807 & &\cellcolor[HTML]{fefefe}-70 &\cellcolor[HTML]{fdfdfe}-159 &\cellcolor[HTML]{fafafd}-398 &\cellcolor[HTML]{f6f5fc}-776 &\cellcolor[HTML]{ecebf9}-1574 &\cellcolor[HTML]{dbd8f5}-3100 \\
isl\_Latn &\cellcolor[HTML]{fefefe}-74 &\cellcolor[HTML]{fdfdfe}-158 &\cellcolor[HTML]{fafafd}-371 &\cellcolor[HTML]{f5f5fc}-799 &\cellcolor[HTML]{eae9f9}-1760 &\cellcolor[HTML]{d1cdf2}-3956 & &\cellcolor[HTML]{fefefe}-66 &\cellcolor[HTML]{fdfdfe}-151 &\cellcolor[HTML]{fbfafd}-336 &\cellcolor[HTML]{f6f5fc}-736 &\cellcolor[HTML]{ebeaf9}-1665 &\cellcolor[HTML]{d5d2f3}-3615 \\
ita\_Latn &\cellcolor[HTML]{fefefe}-18 &\cellcolor[HTML]{fefefe}-56 &\cellcolor[HTML]{fcfcfe}-180 &\cellcolor[HTML]{f8f7fd}-589 &\cellcolor[HTML]{ecebf9}-1579 &\cellcolor[HTML]{ccc9f1}-4322 & &\cellcolor[HTML]{fefefe}-30 &\cellcolor[HTML]{fdfdfe}-122 &\cellcolor[HTML]{fafafd}-349 &\cellcolor[HTML]{f4f3fc}-919 &\cellcolor[HTML]{e4e2f7}-2275 &\cellcolor[HTML]{bcb7ec}-5737 \\
jpn\_Jpan &\cellcolor[HTML]{fefefe}-7 &\cellcolor[HTML]{fefefe}-12 &\cellcolor[HTML]{fefefe}-24 &\cellcolor[HTML]{fefefe}-59 &\cellcolor[HTML]{fdfdfe}-160 &\cellcolor[HTML]{fafafd}-369 & &0 &0 &0 &0 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-5 \\
kan\_Knda &\cellcolor[HTML]{fefefe}-5 &\cellcolor[HTML]{fefefe}-9 &\cellcolor[HTML]{fefefe}-20 &\cellcolor[HTML]{fefefe}-31 &\cellcolor[HTML]{fefefe}-57 &\cellcolor[HTML]{fdfdfe}-129 & &0 &0 &0 &\cellcolor[HTML]{fefefe}-3 &\cellcolor[HTML]{fefefe}-4 &\cellcolor[HTML]{fefefe}-15 \\
kat\_Geor &\cellcolor[HTML]{fefefe}-21 &\cellcolor[HTML]{fefefe}-43 &\cellcolor[HTML]{fefefe}-70 &\cellcolor[HTML]{fdfdfe}-152 &\cellcolor[HTML]{fcfcfe}-241 &\cellcolor[HTML]{faf9fd}-404 & &0 &0 &0 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-15 \\
kaz\_Cyrl &\cellcolor[HTML]{fefefe}-72 &\cellcolor[HTML]{fcfcfe}-213 &\cellcolor[HTML]{f9f8fd}-496 &\cellcolor[HTML]{f2f1fb}-1056 &\cellcolor[HTML]{e4e2f7}-2328 &\cellcolor[HTML]{cbc7f0}-4474 & &\cellcolor[HTML]{fdfdfe}-100 &\cellcolor[HTML]{fcfbfe}-251 &\cellcolor[HTML]{f8f7fd}-589 &\cellcolor[HTML]{f0effa}-1258 &\cellcolor[HTML]{e1dff6}-2536 &\cellcolor[HTML]{c6c2ef}-4867 \\
khk\_Cyrl &\cellcolor[HTML]{fefefe}-72 &\cellcolor[HTML]{fcfcfe}-223 &\cellcolor[HTML]{f8f8fd}-548 &\cellcolor[HTML]{f1f0fb}-1180 &\cellcolor[HTML]{e1dff6}-2537 &\cellcolor[HTML]{c5c1ef}-4966 & &\cellcolor[HTML]{fefefe}-69 &\cellcolor[HTML]{fcfcfe}-213 &\cellcolor[HTML]{f8f8fd}-554 &\cellcolor[HTML]{f0effb}-1247 &\cellcolor[HTML]{e0def6}-2642 &\cellcolor[HTML]{c3bfee}-5135 \\
khm\_Khmr &\cellcolor[HTML]{fefefe}-5 &\cellcolor[HTML]{fefefe}-7 &\cellcolor[HTML]{fefefe}-12 &\cellcolor[HTML]{fefefe}-21 &\cellcolor[HTML]{fefefe}-34 &\cellcolor[HTML]{fefefe}-65 & &0 &0 &0 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-8 &\cellcolor[HTML]{fefefe}-32 \\
kir\_Cyrl &\cellcolor[HTML]{fdfdfe}-100 &\cellcolor[HTML]{fcfcfe}-232 &\cellcolor[HTML]{f8f8fd}-550 &\cellcolor[HTML]{efeefa}-1317 &\cellcolor[HTML]{dfddf6}-2713 &\cellcolor[HTML]{bfbbed}-5455 & &\cellcolor[HTML]{fdfdfe}-109 &\cellcolor[HTML]{fcfbfe}-248 &\cellcolor[HTML]{f7f7fc}-632 &\cellcolor[HTML]{eeecfa}-1463 &\cellcolor[HTML]{dbd8f5}-3069 &\cellcolor[HTML]{b6b1ea}-6264 \\
kor\_Hang &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-5 &\cellcolor[HTML]{fefefe}-8 &\cellcolor[HTML]{fefefe}-13 &\cellcolor[HTML]{fefefe}-28 &\cellcolor[HTML]{fefefe}-66 & &0 &0 &0 &0 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-10 \\
lit\_Latn &\cellcolor[HTML]{fefefe}-55 &\cellcolor[HTML]{fdfdfe}-143 &\cellcolor[HTML]{fafafd}-399 &\cellcolor[HTML]{f3f3fb}-959 &\cellcolor[HTML]{e6e4f8}-2146 &\cellcolor[HTML]{c7c3ef}-4796 & &\cellcolor[HTML]{fefefe}-51 &\cellcolor[HTML]{fdfdfe}-156 &\cellcolor[HTML]{faf9fd}-426 &\cellcolor[HTML]{f2f1fb}-1069 &\cellcolor[HTML]{e3e1f7}-2390 &\cellcolor[HTML]{c2beee}-5220 \\
lvs\_Latn &\cellcolor[HTML]{fefefe}-71 &\cellcolor[HTML]{fdfcfe}-169 &\cellcolor[HTML]{fbfafd}-338 &\cellcolor[HTML]{f5f5fc}-793 &\cellcolor[HTML]{eae8f9}-1788 &\cellcolor[HTML]{d3d0f3}-3718 & &\cellcolor[HTML]{fefefe}-43 &\cellcolor[HTML]{fdfdfe}-159 &\cellcolor[HTML]{fbfafd}-330 &\cellcolor[HTML]{f6f5fc}-775 &\cellcolor[HTML]{eae8f9}-1788 &\cellcolor[HTML]{d4d1f3}-3692 \\
mal\_Mlym &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-4 &\cellcolor[HTML]{fefefe}-14 &\cellcolor[HTML]{fefefe}-36 &\cellcolor[HTML]{fefefe}-67 &\cellcolor[HTML]{fdfdfe}-143 & &0 &0 &\cellcolor[HTML]{fefefe}-4 &\cellcolor[HTML]{fefefe}-5 &\cellcolor[HTML]{fefefe}-11 &\cellcolor[HTML]{fefefe}-16 \\
mar\_Deva &\cellcolor[HTML]{fefefe}-17 &\cellcolor[HTML]{fefefe}-50 &\cellcolor[HTML]{fdfdfe}-116 &\cellcolor[HTML]{fcfbfe}-245 &\cellcolor[HTML]{f9f9fd}-449 &\cellcolor[HTML]{f4f3fc}-926 & &0 &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-3 &\cellcolor[HTML]{fefefe}-6 &\cellcolor[HTML]{fefefe}-12 &\cellcolor[HTML]{fefefe}-29 \\
mkd\_Cyrl &\cellcolor[HTML]{fefefe}-25 &\cellcolor[HTML]{fefefe}-72 &\cellcolor[HTML]{fcfcfe}-208 &\cellcolor[HTML]{f9f9fd}-476 &\cellcolor[HTML]{f1f0fb}-1187 &\cellcolor[HTML]{dfdcf6}-2746 & &\cellcolor[HTML]{fefefe}-22 &\cellcolor[HTML]{fefefe}-71 &\cellcolor[HTML]{fdfcfe}-165 &\cellcolor[HTML]{fafafd}-358 &\cellcolor[HTML]{f5f4fc}-829 &\cellcolor[HTML]{eae9f9}-1754 \\
mya\_Mymr &\cellcolor[HTML]{fefefe}-71 &\cellcolor[HTML]{fdfcfe}-163 &\cellcolor[HTML]{f9f9fd}-435 &\cellcolor[HTML]{f1f1fb}-1127 &\cellcolor[HTML]{e0def6}-2617 &\cellcolor[HTML]{beb9ed}-5592 & &0 &0 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-4 &\cellcolor[HTML]{fefefe}-11 &\cellcolor[HTML]{fefefe}-29 \\
nld\_Latn &\cellcolor[HTML]{fefefe}-38 &\cellcolor[HTML]{fdfdfe}-106 &\cellcolor[HTML]{fbfbfe}-289 &\cellcolor[HTML]{f7f6fc}-677 &\cellcolor[HTML]{eeedfa}-1440 &\cellcolor[HTML]{dad8f4}-3137 & &\cellcolor[HTML]{fefefe}-59 &\cellcolor[HTML]{fdfdfe}-142 &\cellcolor[HTML]{fafafd}-356 &\cellcolor[HTML]{f5f5fc}-804 &\cellcolor[HTML]{ecebf9}-1584 &\cellcolor[HTML]{dad7f4}-3169 \\
nno\_Latn &\cellcolor[HTML]{fefefe}-41 &\cellcolor[HTML]{fdfdfe}-91 &\cellcolor[HTML]{fcfcfe}-229 &\cellcolor[HTML]{f9f9fd}-476 &\cellcolor[HTML]{f1f0fb}-1130 &\cellcolor[HTML]{dfddf6}-2711 & &\cellcolor[HTML]{fefefe}-43 &\cellcolor[HTML]{fdfdfe}-106 &\cellcolor[HTML]{fcfbfe}-257 &\cellcolor[HTML]{f8f7fd}-571 &\cellcolor[HTML]{f0effb}-1238 &\cellcolor[HTML]{dfddf6}-2679 \\
nob\_Latn &\cellcolor[HTML]{fefefe}-35 &\cellcolor[HTML]{fefdfe}-81 &\cellcolor[HTML]{fcfcfe}-203 &\cellcolor[HTML]{f9f9fd}-432 &\cellcolor[HTML]{f3f2fb}-968 &\cellcolor[HTML]{e6e4f8}-2148 & &\cellcolor[HTML]{fefefe}-35 &\cellcolor[HTML]{fdfdfe}-108 &\cellcolor[HTML]{fcfbfe}-249 &\cellcolor[HTML]{f8f8fd}-556 &\cellcolor[HTML]{f2f1fb}-1116 &\cellcolor[HTML]{e4e2f7}-2302 \\
npi\_Deva &\cellcolor[HTML]{fefefe}-30 &\cellcolor[HTML]{fefefe}-53 &\cellcolor[HTML]{fdfdfe}-110 &\cellcolor[HTML]{fcfcfe}-197 &\cellcolor[HTML]{fafafd}-380 &\cellcolor[HTML]{f5f5fc}-800 & &0 &0 &0 &0 &0 &\cellcolor[HTML]{fefefe}-2 \\
ory\_Orya &0 &0 &0 &0 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-8 & &0 &0 &0 &0 &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-7 \\
pan\_Guru &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-6 &\cellcolor[HTML]{fefefe}-5 &\cellcolor[HTML]{fefefe}-8 &\cellcolor[HTML]{fefefe}-23 &\cellcolor[HTML]{fefefe}-40 & &0 &0 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-11 &\cellcolor[HTML]{fefefe}-33 \\
pol\_Latn &\cellcolor[HTML]{fefefe}-39 &\cellcolor[HTML]{fdfdfe}-127 &\cellcolor[HTML]{fbfafd}-324 &\cellcolor[HTML]{f5f4fc}-856 &\cellcolor[HTML]{e3e1f7}-2388 &\cellcolor[HTML]{b8b3eb}-6054 & &\cellcolor[HTML]{fefefe}-66 &\cellcolor[HTML]{fcfcfe}-198 &\cellcolor[HTML]{f8f7fd}-576 &\cellcolor[HTML]{eeedfa}-1400 &\cellcolor[HTML]{d7d4f4}-3410 &\cellcolor[HTML]{a59fe6}-7708 \\
por\_Latn &\cellcolor[HTML]{fefefe}-13 &\cellcolor[HTML]{fefefe}-38 &\cellcolor[HTML]{fdfdfe}-126 &\cellcolor[HTML]{fbfbfe}-292 &\cellcolor[HTML]{f7f6fc}-663 &\cellcolor[HTML]{ecebf9}-1594 & &\cellcolor[HTML]{fefefe}-25 &\cellcolor[HTML]{fefefe}-78 &\cellcolor[HTML]{fcfcfe}-206 &\cellcolor[HTML]{f8f8fd}-518 &\cellcolor[HTML]{f1f0fb}-1149 &\cellcolor[HTML]{e2e0f7}-2424 \\
ron\_Latn &\cellcolor[HTML]{fefefe}-36 &\cellcolor[HTML]{fdfdfe}-101 &\cellcolor[HTML]{fcfbfe}-258 &\cellcolor[HTML]{f7f6fc}-672 &\cellcolor[HTML]{ecebf9}-1580 &\cellcolor[HTML]{d4d1f3}-3697 & &\cellcolor[HTML]{fefefe}-41 &\cellcolor[HTML]{fdfdfe}-127 &\cellcolor[HTML]{fafafd}-355 &\cellcolor[HTML]{f4f4fc}-874 &\cellcolor[HTML]{e6e4f8}-2133 &\cellcolor[HTML]{c7c3ef}-4756 \\
rus\_Cyrl &\cellcolor[HTML]{fefefe}-19 &\cellcolor[HTML]{fefefe}-56 &\cellcolor[HTML]{fcfcfe}-187 &\cellcolor[HTML]{f9f9fd}-467 &\cellcolor[HTML]{f1f0fb}-1155 &\cellcolor[HTML]{dedcf6}-2795 & &\cellcolor[HTML]{fefefe}-17 &\cellcolor[HTML]{fefefe}-45 &\cellcolor[HTML]{fdfdfe}-101 &\cellcolor[HTML]{fbfbfe}-270 &\cellcolor[HTML]{f8f7fd}-573 &\cellcolor[HTML]{f0effa}-1281 \\
sin\_Sinh &0 &0 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-4 & &0 &0 &0 &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-3 &\cellcolor[HTML]{fefefe}-10 \\
slk\_Latn &\cellcolor[HTML]{fefefe}-43 &\cellcolor[HTML]{fdfdfe}-130 &\cellcolor[HTML]{fbfbfd}-313 &\cellcolor[HTML]{f5f5fc}-786 &\cellcolor[HTML]{e9e7f9}-1864 &\cellcolor[HTML]{ccc9f1}-4326 & &\cellcolor[HTML]{fefefe}-45 &\cellcolor[HTML]{fdfdfe}-134 &\cellcolor[HTML]{fafafd}-380 &\cellcolor[HTML]{f3f3fb}-955 &\cellcolor[HTML]{e5e3f7}-2237 &\cellcolor[HTML]{c5c1ef}-4932 \\
slv\_Latn &\cellcolor[HTML]{fefefe}-44 &\cellcolor[HTML]{fdfdfe}-125 &\cellcolor[HTML]{fafafd}-373 &\cellcolor[HTML]{f3f2fb}-1019 &\cellcolor[HTML]{e0def6}-2612 &\cellcolor[HTML]{b5b0ea}-6297 & &\cellcolor[HTML]{fefefe}-63 &\cellcolor[HTML]{fcfcfe}-198 &\cellcolor[HTML]{f8f8fd}-538 &\cellcolor[HTML]{eeedfa}-1443 &\cellcolor[HTML]{d6d3f3}-3468 &\cellcolor[HTML]{a59fe6}-7712 \\
spa\_Latn &\cellcolor[HTML]{fefefe}-6 &\cellcolor[HTML]{fefefe}-18 &\cellcolor[HTML]{fefefe}-69 &\cellcolor[HTML]{fcfcfe}-207 &\cellcolor[HTML]{f7f7fc}-628 &\cellcolor[HTML]{e9e8f9}-1834 & &\cellcolor[HTML]{fefefe}-24 &\cellcolor[HTML]{fefefe}-73 &\cellcolor[HTML]{fcfcfe}-213 &\cellcolor[HTML]{f8f8fd}-524 &\cellcolor[HTML]{f0effa}-1276 &\cellcolor[HTML]{dcd9f5}-3009 \\
srp\_Cyrl &\cellcolor[HTML]{fefefe}-37 &\cellcolor[HTML]{fdfdfe}-127 &\cellcolor[HTML]{faf9fd}-406 &\cellcolor[HTML]{f1f0fb}-1146 &\cellcolor[HTML]{dddaf5}-2927 &\cellcolor[HTML]{aea8e8}-6961 & &\cellcolor[HTML]{fefefe}-18 &\cellcolor[HTML]{fefefe}-58 &\cellcolor[HTML]{fdfdfe}-122 &\cellcolor[HTML]{fbfbfe}-300 &\cellcolor[HTML]{f6f5fc}-765 &\cellcolor[HTML]{eae8f9}-1805 \\
swe\_Latn &\cellcolor[HTML]{fefefe}-71 &\cellcolor[HTML]{fdfdfe}-135 &\cellcolor[HTML]{fbfbfe}-291 &\cellcolor[HTML]{f8f7fd}-566 &\cellcolor[HTML]{f1f0fb}-1137 &\cellcolor[HTML]{e1dff6}-2516 & &\cellcolor[HTML]{fefefe}-64 &\cellcolor[HTML]{fdfdfe}-135 &\cellcolor[HTML]{fbfbfe}-287 &\cellcolor[HTML]{f8f7fd}-589 &\cellcolor[HTML]{f1f1fb}-1127 &\cellcolor[HTML]{e4e2f7}-2282 \\
swh\_Latn &\cellcolor[HTML]{fdfdfe}-125 &\cellcolor[HTML]{fafafd}-356 &\cellcolor[HTML]{f3f2fb}-983 &\cellcolor[HTML]{e5e3f7}-2228 &\cellcolor[HTML]{c4c0ee}-5034 &\cellcolor[HTML]{7f76db}-10967 & &\cellcolor[HTML]{fdfdfe}-160 &\cellcolor[HTML]{f9f9fd}-458 &\cellcolor[HTML]{f1f0fb}-1199 &\cellcolor[HTML]{dfddf6}-2712 &\cellcolor[HTML]{bdb8ec}-5683 &\cellcolor[HTML]{746ad8}-11956 \\
tam\_Taml &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-11 &\cellcolor[HTML]{fefefe}-28 &\cellcolor[HTML]{fefefe}-69 &\cellcolor[HTML]{fdfdfe}-151 &\cellcolor[HTML]{fcfcfe}-237 & &0 &0 &0 &0 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-7 \\
tel\_Telu &\cellcolor[HTML]{fefefe}-18 &\cellcolor[HTML]{fefefe}-36 &\cellcolor[HTML]{fefefe}-78 &\cellcolor[HTML]{fcfcfe}-181 &\cellcolor[HTML]{fbfafd}-335 &\cellcolor[HTML]{f7f7fc}-631 & &0 &0 &0 &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-2 &\cellcolor[HTML]{fefefe}-17 \\
tha\_Thai &\cellcolor[HTML]{fcfbfe}-250 &\cellcolor[HTML]{f8f7fd}-567 &\cellcolor[HTML]{efeefa}-1319 &\cellcolor[HTML]{dedcf6}-2768 &\cellcolor[HTML]{bab5ec}-5888 &\cellcolor[HTML]{7268d8}-12112 & &0 &0 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-3 &\cellcolor[HTML]{fefefe}-14 &\cellcolor[HTML]{fefefe}-38 \\
tur\_Latn &\cellcolor[HTML]{fefefe}-49 &\cellcolor[HTML]{fdfdfe}-115 &\cellcolor[HTML]{fbfbfe}-285 &\cellcolor[HTML]{f7f6fc}-673 &\cellcolor[HTML]{ebeaf9}-1662 &\cellcolor[HTML]{d4d1f3}-3679 & &\cellcolor[HTML]{fdfdfe}-91 &\cellcolor[HTML]{fcfcfe}-201 &\cellcolor[HTML]{f9f9fd}-438 &\cellcolor[HTML]{f3f3fb}-953 &\cellcolor[HTML]{e5e3f7}-2229 &\cellcolor[HTML]{c8c4ef}-4705 \\
ukr\_Cyrl &\cellcolor[HTML]{fefefe}-31 &\cellcolor[HTML]{fdfdfe}-92 &\cellcolor[HTML]{fbfbfe}-302 &\cellcolor[HTML]{f5f5fc}-793 &\cellcolor[HTML]{e9e7f9}-1862 &\cellcolor[HTML]{cdcaf1}-4242 & &\cellcolor[HTML]{fefefe}-22 &\cellcolor[HTML]{fefefe}-49 &\cellcolor[HTML]{fdfdfe}-113 &\cellcolor[HTML]{fbfbfe}-299 &\cellcolor[HTML]{f7f7fd}-620 &\cellcolor[HTML]{efeefa}-1324 \\
urd\_Arab &\cellcolor[HTML]{fefefe}-31 &\cellcolor[HTML]{fefefe}-78 &\cellcolor[HTML]{fcfcfe}-177 &\cellcolor[HTML]{fafafd}-392 &\cellcolor[HTML]{f5f5fc}-781 &\cellcolor[HTML]{eeedfa}-1392 & &0 &0 &0 &\cellcolor[HTML]{fefefe}-1 &\cellcolor[HTML]{fefefe}-4 &\cellcolor[HTML]{fefefe}-37 \\
uzn\_Latn &\cellcolor[HTML]{fefefe}-64 &\cellcolor[HTML]{fcfcfe}-214 &\cellcolor[HTML]{f8f8fd}-556 &\cellcolor[HTML]{efeefa}-1316 &\cellcolor[HTML]{dddaf5}-2911 &\cellcolor[HTML]{b4afea}-6400 & &\cellcolor[HTML]{fdfdfe}-128 &\cellcolor[HTML]{fbfbfe}-302 &\cellcolor[HTML]{f6f5fc}-766 &\cellcolor[HTML]{ebeaf9}-1647 &\cellcolor[HTML]{d6d3f3}-3490 &\cellcolor[HTML]{afaae9}-6834 \\
vie\_Latn &\cellcolor[HTML]{fefefe}-5 &\cellcolor[HTML]{fefefe}-7 &\cellcolor[HTML]{fefefe}-16 &\cellcolor[HTML]{fdfdfe}-98 &\cellcolor[HTML]{fbfbfe}-281 &\cellcolor[HTML]{f5f4fc}-851 & &\cellcolor[HTML]{fefefe}-13 &\cellcolor[HTML]{fefefe}-32 &\cellcolor[HTML]{fefefe}-59 &\cellcolor[HTML]{fdfdfe}-115 &\cellcolor[HTML]{fbfafd}-333 &\cellcolor[HTML]{f3f2fb}-991 \\
zsm\_Latn &\cellcolor[HTML]{fefefe}-49 &\cellcolor[HTML]{fdfdfe}-118 &\cellcolor[HTML]{fbfbfe}-302 &\cellcolor[HTML]{f7f6fc}-658 &\cellcolor[HTML]{eeedfa}-1412 &\cellcolor[HTML]{dedbf5}-2821 & &\cellcolor[HTML]{fefefe}-48 &\cellcolor[HTML]{fdfdfe}-136 &\cellcolor[HTML]{fbfafd}-342 &\cellcolor[HTML]{f6f5fc}-767 &\cellcolor[HTML]{ecebf9}-1602 &\cellcolor[HTML]{dbd8f5}-3063 \\
\bottomrule
\end{tabular}

\begin{tabular}{lcccccccccccccccc}\toprule
&\multicolumn{7}{c}{\textbf{Llama-3 extension}} & &\multicolumn{7}{c}{\textbf{Qwen-2.5 extension}} \\\cmidrule{2-8}\cmidrule{10-16}
\textbf{Lang.} &\textbf{0} &\textbf{1000} &\textbf{2000} &\textbf{4000} &\textbf{8000} &\textbf{16000} &\textbf{32000} & &\textbf{0} &\textbf{1000} &\textbf{2000} &\textbf{4000} &\textbf{8000} &\textbf{16000} &\textbf{32000} \\\midrule
afr\_Latn &\cellcolor[HTML]{eceafa}3.093 &\cellcolor[HTML]{e6e5f9}3.630 &\cellcolor[HTML]{e5e3f8}3.828 &\cellcolor[HTML]{e2e0f7}4.062 &\cellcolor[HTML]{e0def7}4.301 &\cellcolor[HTML]{dedbf6}4.529 &\cellcolor[HTML]{dcdaf6}4.720 & &\cellcolor[HTML]{ecebfa}3.051 &\cellcolor[HTML]{e7e5f9}3.579 &\cellcolor[HTML]{e5e3f8}3.773 &\cellcolor[HTML]{e3e1f8}4.004 &\cellcolor[HTML]{e1def7}4.239 &\cellcolor[HTML]{dedcf6}4.463 &\cellcolor[HTML]{dddaf6}4.649 \\
als\_Latn &\cellcolor[HTML]{f0effb}2.625 &\cellcolor[HTML]{e7e5f9}3.550 &\cellcolor[HTML]{e5e3f8}3.770 &\cellcolor[HTML]{e3e0f7}4.036 &\cellcolor[HTML]{e0def7}4.328 &\cellcolor[HTML]{dddaf6}4.645 &\cellcolor[HTML]{dad7f5}4.936 & &\cellcolor[HTML]{f0effb}2.579 &\cellcolor[HTML]{e8e6f9}3.484 &\cellcolor[HTML]{e6e4f8}3.710 &\cellcolor[HTML]{e3e1f8}3.976 &\cellcolor[HTML]{e0def7}4.266 &\cellcolor[HTML]{dddbf6}4.577 &\cellcolor[HTML]{dbd8f5}4.866 \\
arb\_Arab &\cellcolor[HTML]{dcdaf6}4.705 &\cellcolor[HTML]{d8d5f5}5.138 &\cellcolor[HTML]{d5d2f4}5.438 &\cellcolor[HTML]{d1cef3}5.867 &\cellcolor[HTML]{ccc9f1}6.385 &\cellcolor[HTML]{c7c3f0}6.941 &\cellcolor[HTML]{c1bdee}7.542 & &\cellcolor[HTML]{dcdaf6}4.697 &\cellcolor[HTML]{d8d5f5}5.158 &\cellcolor[HTML]{d5d2f4}5.431 &\cellcolor[HTML]{d1cef3}5.838 &\cellcolor[HTML]{cdc9f1}6.313 &\cellcolor[HTML]{c8c4f0}6.855 &\cellcolor[HTML]{c2beef}7.438 \\
ary\_Arab &\cellcolor[HTML]{dfdcf6}4.450 &\cellcolor[HTML]{dbd9f5}4.813 &\cellcolor[HTML]{d9d6f5}5.034 &\cellcolor[HTML]{d6d3f4}5.349 &\cellcolor[HTML]{d2cff3}5.775 &\cellcolor[HTML]{cdcaf2}6.269 &\cellcolor[HTML]{c8c4f0}6.797 & &\cellcolor[HTML]{dfddf7}4.405 &\cellcolor[HTML]{dbd9f5}4.803 &\cellcolor[HTML]{d9d6f5}5.027 &\cellcolor[HTML]{d6d3f4}5.316 &\cellcolor[HTML]{d3cff3}5.711 &\cellcolor[HTML]{cecaf2}6.193 &\cellcolor[HTML]{c9c5f0}6.704 \\
azj\_Latn &\cellcolor[HTML]{f1f0fb}2.537 &\cellcolor[HTML]{e5e3f8}3.787 &\cellcolor[HTML]{e2e0f7}4.120 &\cellcolor[HTML]{dedcf6}4.516 &\cellcolor[HTML]{dad7f5}4.936 &\cellcolor[HTML]{d6d3f4}5.386 &\cellcolor[HTML]{d1cef3}5.835 & &\cellcolor[HTML]{f2f1fc}2.420 &\cellcolor[HTML]{e6e5f9}3.624 &\cellcolor[HTML]{e3e1f8}3.966 &\cellcolor[HTML]{dfddf7}4.388 &\cellcolor[HTML]{dbd8f5}4.834 &\cellcolor[HTML]{d7d4f4}5.281 &\cellcolor[HTML]{d2cff3}5.730 \\
bel\_Cyrl &\cellcolor[HTML]{e4e2f8}3.866 &\cellcolor[HTML]{d7d5f4}5.204 &\cellcolor[HTML]{d3d0f3}5.640 &\cellcolor[HTML]{cecbf2}6.184 &\cellcolor[HTML]{c8c4f0}6.845 &\cellcolor[HTML]{c1bdee}7.547 &\cellcolor[HTML]{bab5ec}8.265 & &\cellcolor[HTML]{eae8f9}3.301 &\cellcolor[HTML]{dbd8f5}4.875 &\cellcolor[HTML]{d6d3f4}5.368 &\cellcolor[HTML]{d0cdf2}5.962 &\cellcolor[HTML]{cac6f1}6.658 &\cellcolor[HTML]{c2beef}7.402 &\cellcolor[HTML]{bbb7ed}8.140 \\
ben\_Beng &\cellcolor[HTML]{f4f3fc}2.203 &\cellcolor[HTML]{d9d6f5}5.023 &\cellcolor[HTML]{d7d5f4}5.196 &\cellcolor[HTML]{d7d4f4}5.291 &\cellcolor[HTML]{d6d3f4}5.345 &\cellcolor[HTML]{d6d3f4}5.366 &\cellcolor[HTML]{d6d3f4}5.375 & &\cellcolor[HTML]{f1f0fc}2.483 &\cellcolor[HTML]{d8d5f5}5.159 &\cellcolor[HTML]{d6d3f4}5.336 &\cellcolor[HTML]{d5d2f4}5.429 &\cellcolor[HTML]{d5d2f4}5.482 &\cellcolor[HTML]{d5d2f4}5.503 &\cellcolor[HTML]{d4d1f4}5.513 \\
bos\_Latn &\cellcolor[HTML]{efedfb}2.780 &\cellcolor[HTML]{eae8fa}3.265 &\cellcolor[HTML]{e8e6f9}3.476 &\cellcolor[HTML]{e6e4f8}3.714 &\cellcolor[HTML]{e3e1f8}3.998 &\cellcolor[HTML]{e0ddf7}4.330 &\cellcolor[HTML]{dddaf6}4.663 & &\cellcolor[HTML]{f0effb}2.651 &\cellcolor[HTML]{ebe9fa}3.161 &\cellcolor[HTML]{e9e7f9}3.377 &\cellcolor[HTML]{e6e5f9}3.635 &\cellcolor[HTML]{e4e2f8}3.920 &\cellcolor[HTML]{e0def7}4.255 &\cellcolor[HTML]{dddbf6}4.588 \\
bul\_Cyrl &\cellcolor[HTML]{dedbf6}4.527 &\cellcolor[HTML]{d2cff3}5.746 &\cellcolor[HTML]{cfccf2}6.087 &\cellcolor[HTML]{cbc7f1}6.511 &\cellcolor[HTML]{c6c2f0}7.054 &\cellcolor[HTML]{c0bbee}7.675 &\cellcolor[HTML]{bab5ec}8.286 & &\cellcolor[HTML]{e2e0f7}4.115 &\cellcolor[HTML]{d6d3f4}5.395 &\cellcolor[HTML]{d2cff3}5.783 &\cellcolor[HTML]{cdcaf2}6.258 &\cellcolor[HTML]{c8c4f0}6.856 &\cellcolor[HTML]{c1bdee}7.511 &\cellcolor[HTML]{bbb7ed}8.139 \\
cat\_Latn &\cellcolor[HTML]{eae9fa}3.219 &\cellcolor[HTML]{e6e4f8}3.705 &\cellcolor[HTML]{e4e2f8}3.865 &\cellcolor[HTML]{e2e0f7}4.066 &\cellcolor[HTML]{e0def7}4.291 &\cellcolor[HTML]{dedbf6}4.530 &\cellcolor[HTML]{dcd9f6}4.721 & &\cellcolor[HTML]{ebe9fa}3.194 &\cellcolor[HTML]{e6e4f8}3.662 &\cellcolor[HTML]{e5e3f8}3.816 &\cellcolor[HTML]{e3e1f8}4.017 &\cellcolor[HTML]{e1def7}4.236 &\cellcolor[HTML]{dedcf6}4.473 &\cellcolor[HTML]{dddaf6}4.657 \\
ces\_Latn &\cellcolor[HTML]{e8e6f9}3.485 &\cellcolor[HTML]{e6e4f8}3.700 &\cellcolor[HTML]{e5e3f8}3.827 &\cellcolor[HTML]{e3e1f8}4.013 &\cellcolor[HTML]{e0def7}4.286 &\cellcolor[HTML]{dddaf6}4.631 &\cellcolor[HTML]{d9d6f5}5.020 & &\cellcolor[HTML]{f1f0fc}2.496 &\cellcolor[HTML]{ebe9fa}3.193 &\cellcolor[HTML]{e8e6f9}3.449 &\cellcolor[HTML]{e5e3f8}3.743 &\cellcolor[HTML]{e2e0f7}4.081 &\cellcolor[HTML]{dedcf6}4.483 &\cellcolor[HTML]{dad8f5}4.897 \\
cmn\_Hani &\cellcolor[HTML]{e8e6f9}3.471 &\cellcolor[HTML]{e6e4f9}3.651 &\cellcolor[HTML]{e5e3f8}3.788 &\cellcolor[HTML]{e3e1f8}3.980 &\cellcolor[HTML]{e1def7}4.250 &\cellcolor[HTML]{dedbf6}4.559 &\cellcolor[HTML]{dad8f5}4.903 & &\cellcolor[HTML]{dfddf7}4.361 &\cellcolor[HTML]{dedcf6}4.493 &\cellcolor[HTML]{dedbf6}4.548 &\cellcolor[HTML]{dddbf6}4.614 &\cellcolor[HTML]{dcdaf6}4.706 &\cellcolor[HTML]{dbd8f5}4.843 &\cellcolor[HTML]{d9d6f5}5.032 \\
dan\_Latn &\cellcolor[HTML]{ebe9fa}3.157 &\cellcolor[HTML]{e7e5f9}3.579 &\cellcolor[HTML]{e5e3f8}3.752 &\cellcolor[HTML]{e3e1f8}3.986 &\cellcolor[HTML]{e1dff7}4.230 &\cellcolor[HTML]{dedcf6}4.483 &\cellcolor[HTML]{dcd9f6}4.729 & &\cellcolor[HTML]{ebeafa}3.112 &\cellcolor[HTML]{e7e6f9}3.527 &\cellcolor[HTML]{e6e4f8}3.696 &\cellcolor[HTML]{e4e2f8}3.925 &\cellcolor[HTML]{e1dff7}4.163 &\cellcolor[HTML]{dfddf6}4.414 &\cellcolor[HTML]{dddaf6}4.654 \\
deu\_Latn &\cellcolor[HTML]{e6e5f9}3.638 &\cellcolor[HTML]{e4e2f8}3.877 &\cellcolor[HTML]{e2e0f7}4.047 &\cellcolor[HTML]{e0def7}4.259 &\cellcolor[HTML]{dedbf6}4.529 &\cellcolor[HTML]{dbd8f5}4.828 &\cellcolor[HTML]{d8d5f5}5.115 & &\cellcolor[HTML]{e6e5f9}3.631 &\cellcolor[HTML]{e4e2f8}3.855 &\cellcolor[HTML]{e3e1f8}4.012 &\cellcolor[HTML]{e1dff7}4.215 &\cellcolor[HTML]{dedcf6}4.476 &\cellcolor[HTML]{dcd9f6}4.763 &\cellcolor[HTML]{d9d6f5}5.042 \\
ekk\_Latn &\cellcolor[HTML]{f0effb}2.644 &\cellcolor[HTML]{eae8f9}3.282 &\cellcolor[HTML]{e8e6f9}3.509 &\cellcolor[HTML]{e5e3f8}3.767 &\cellcolor[HTML]{e2e0f7}4.084 &\cellcolor[HTML]{dfdcf6}4.441 &\cellcolor[HTML]{dbd9f5}4.798 & &\cellcolor[HTML]{f0effb}2.607 &\cellcolor[HTML]{eae9fa}3.230 &\cellcolor[HTML]{e8e6f9}3.454 &\cellcolor[HTML]{e6e4f8}3.711 &\cellcolor[HTML]{e3e1f8}4.023 &\cellcolor[HTML]{dfddf7}4.370 &\cellcolor[HTML]{dcdaf6}4.717 \\
ell\_Grek &\cellcolor[HTML]{dcd9f6}4.736 &\cellcolor[HTML]{d5d2f4}5.443 &\cellcolor[HTML]{d1cef3}5.841 &\cellcolor[HTML]{ccc8f1}6.433 &\cellcolor[HTML]{c5c1ef}7.148 &\cellcolor[HTML]{bdb9ed}7.924 &\cellcolor[HTML]{b6b1eb}8.683 & &\cellcolor[HTML]{f5f4fd}2.123 &\cellcolor[HTML]{dddaf6}4.647 &\cellcolor[HTML]{d6d3f4}5.317 &\cellcolor[HTML]{cfccf2}6.054 &\cellcolor[HTML]{c7c3f0}6.895 &\cellcolor[HTML]{bfbbee}7.748 &\cellcolor[HTML]{b8b3ec}8.538 \\
eus\_Latn &\cellcolor[HTML]{efedfb}2.774 &\cellcolor[HTML]{e7e5f9}3.581 &\cellcolor[HTML]{e4e2f8}3.851 &\cellcolor[HTML]{e1dff7}4.174 &\cellcolor[HTML]{dedcf6}4.516 &\cellcolor[HTML]{dad8f5}4.890 &\cellcolor[HTML]{d7d4f4}5.247 & &\cellcolor[HTML]{efeefb}2.722 &\cellcolor[HTML]{e7e6f9}3.519 &\cellcolor[HTML]{e5e3f8}3.787 &\cellcolor[HTML]{e2e0f7}4.106 &\cellcolor[HTML]{dfdcf6}4.443 &\cellcolor[HTML]{dbd9f5}4.810 &\cellcolor[HTML]{d8d5f5}5.158 \\
fas\_Arab &\cellcolor[HTML]{d6d3f4}5.404 &\cellcolor[HTML]{d2cff3}5.769 &\cellcolor[HTML]{d0ccf2}6.030 &\cellcolor[HTML]{ccc9f1}6.377 &\cellcolor[HTML]{c9c5f0}6.755 &\cellcolor[HTML]{c6c2f0}7.063 &\cellcolor[HTML]{c4bfef}7.284 & &\cellcolor[HTML]{ebeafa}3.140 &\cellcolor[HTML]{dcdaf6}4.705 &\cellcolor[HTML]{d8d5f4}5.191 &\cellcolor[HTML]{d2cff3}5.753 &\cellcolor[HTML]{cdc9f2}6.297 &\cellcolor[HTML]{c9c5f1}6.683 &\cellcolor[HTML]{c7c3f0}6.914 \\
fin\_Latn &\cellcolor[HTML]{efeefb}2.704 &\cellcolor[HTML]{e9e7f9}3.405 &\cellcolor[HTML]{e6e4f8}3.661 &\cellcolor[HTML]{e3e1f8}3.957 &\cellcolor[HTML]{e0def7}4.323 &\cellcolor[HTML]{dcd9f6}4.732 &\cellcolor[HTML]{d8d5f5}5.158 & &\cellcolor[HTML]{f0eefb}2.674 &\cellcolor[HTML]{e9e7f9}3.359 &\cellcolor[HTML]{e7e5f9}3.611 &\cellcolor[HTML]{e4e2f8}3.901 &\cellcolor[HTML]{e0def7}4.258 &\cellcolor[HTML]{dddaf6}4.657 &\cellcolor[HTML]{d9d6f5}5.072 \\
fra\_Latn &\cellcolor[HTML]{e5e3f8}3.771 &\cellcolor[HTML]{e3e1f8}4.025 &\cellcolor[HTML]{e1dff7}4.172 &\cellcolor[HTML]{dfddf7}4.363 &\cellcolor[HTML]{dddbf6}4.596 &\cellcolor[HTML]{dbd8f5}4.820 &\cellcolor[HTML]{d9d7f5}5.007 & &\cellcolor[HTML]{e5e3f8}3.756 &\cellcolor[HTML]{e3e1f8}3.996 &\cellcolor[HTML]{e2dff7}4.134 &\cellcolor[HTML]{e0def7}4.313 &\cellcolor[HTML]{dedbf6}4.534 &\cellcolor[HTML]{dcd9f6}4.749 &\cellcolor[HTML]{dad7f5}4.929 \\
glg\_Latn &\cellcolor[HTML]{e7e6f9}3.536 &\cellcolor[HTML]{e4e2f8}3.924 &\cellcolor[HTML]{e2e0f7}4.084 &\cellcolor[HTML]{e0def7}4.310 &\cellcolor[HTML]{dedbf6}4.558 &\cellcolor[HTML]{dbd9f5}4.818 &\cellcolor[HTML]{d9d6f5}5.042 & &\cellcolor[HTML]{e8e6f9}3.513 &\cellcolor[HTML]{e4e2f8}3.894 &\cellcolor[HTML]{e2e0f7}4.042 &\cellcolor[HTML]{e0def7}4.262 &\cellcolor[HTML]{dedcf6}4.497 &\cellcolor[HTML]{dcd9f6}4.750 &\cellcolor[HTML]{dad7f5}4.967 \\
guj\_Gujr &\cellcolor[HTML]{faf9fe}1.590 &\cellcolor[HTML]{d9d6f5}5.019 &\cellcolor[HTML]{d7d4f4}5.226 &\cellcolor[HTML]{d6d3f4}5.340 &\cellcolor[HTML]{d6d3f4}5.401 &\cellcolor[HTML]{d5d2f4}5.429 &\cellcolor[HTML]{d5d2f4}5.439 & &\cellcolor[HTML]{f8f8fd}1.779 &\cellcolor[HTML]{d9d7f5}4.994 &\cellcolor[HTML]{d8d5f4}5.193 &\cellcolor[HTML]{d6d4f4}5.302 &\cellcolor[HTML]{d6d3f4}5.361 &\cellcolor[HTML]{d6d3f4}5.390 &\cellcolor[HTML]{d6d3f4}5.400 \\
heb\_Hebr &\cellcolor[HTML]{f7f7fd}1.849 &\cellcolor[HTML]{e1dff7}4.150 &\cellcolor[HTML]{dddaf6}4.655 &\cellcolor[HTML]{d7d4f4}5.220 &\cellcolor[HTML]{d2cef3}5.818 &\cellcolor[HTML]{ccc8f1}6.442 &\cellcolor[HTML]{c6c1ef}7.079 & &\cellcolor[HTML]{dedcf6}4.497 &\cellcolor[HTML]{dad7f5}4.949 &\cellcolor[HTML]{d7d5f4}5.197 &\cellcolor[HTML]{d4d1f4}5.540 &\cellcolor[HTML]{d0cdf2}5.959 &\cellcolor[HTML]{cbc8f1}6.456 &\cellcolor[HTML]{c6c2f0}7.007 \\
hin\_Deva &\cellcolor[HTML]{dad7f5}4.920 &\cellcolor[HTML]{d9d6f5}5.089 &\cellcolor[HTML]{d8d5f5}5.139 &\cellcolor[HTML]{d8d5f4}5.175 &\cellcolor[HTML]{d7d5f4}5.195 &\cellcolor[HTML]{d7d5f4}5.206 &\cellcolor[HTML]{d7d5f4}5.209 & &\cellcolor[HTML]{efedfb}2.764 &\cellcolor[HTML]{dad8f5}4.902 &\cellcolor[HTML]{d9d6f5}5.064 &\cellcolor[HTML]{d8d5f5}5.130 &\cellcolor[HTML]{d8d5f5}5.161 &\cellcolor[HTML]{d8d5f4}5.173 &\cellcolor[HTML]{d8d5f4}5.177 \\
hrv\_Latn &\cellcolor[HTML]{efeefb}2.757 &\cellcolor[HTML]{eae8fa}3.263 &\cellcolor[HTML]{e8e6f9}3.462 &\cellcolor[HTML]{e6e4f8}3.723 &\cellcolor[HTML]{e3e1f8}4.022 &\cellcolor[HTML]{dfddf7}4.360 &\cellcolor[HTML]{dcdaf6}4.720 & &\cellcolor[HTML]{f0effb}2.629 &\cellcolor[HTML]{ebe9fa}3.157 &\cellcolor[HTML]{e9e7f9}3.372 &\cellcolor[HTML]{e6e5f9}3.638 &\cellcolor[HTML]{e3e1f8}3.944 &\cellcolor[HTML]{e0def7}4.280 &\cellcolor[HTML]{dddaf6}4.639 \\
hun\_Latn &\cellcolor[HTML]{f0effb}2.655 &\cellcolor[HTML]{eae8f9}3.302 &\cellcolor[HTML]{e7e5f9}3.581 &\cellcolor[HTML]{e4e2f8}3.913 &\cellcolor[HTML]{e0def7}4.304 &\cellcolor[HTML]{dcdaf6}4.709 &\cellcolor[HTML]{d8d5f5}5.135 & &\cellcolor[HTML]{f0effb}2.605 &\cellcolor[HTML]{eae9fa}3.247 &\cellcolor[HTML]{e7e6f9}3.524 &\cellcolor[HTML]{e4e2f8}3.852 &\cellcolor[HTML]{e1def7}4.239 &\cellcolor[HTML]{dddaf6}4.638 &\cellcolor[HTML]{d9d6f5}5.051 \\
hye\_Armn &\cellcolor[HTML]{fefeff}1.147 &\cellcolor[HTML]{dad7f5}4.950 &\cellcolor[HTML]{d2cff3}5.753 &\cellcolor[HTML]{cac6f1}6.603 &\cellcolor[HTML]{c2bdee}7.472 &\cellcolor[HTML]{bab5ec}8.315 &\cellcolor[HTML]{b2acea}9.146 & &\cellcolor[HTML]{f7f7fd}1.851 &\cellcolor[HTML]{d9d6f5}5.040 &\cellcolor[HTML]{d2cff3}5.765 &\cellcolor[HTML]{cac7f1}6.580 &\cellcolor[HTML]{c2beef}7.412 &\cellcolor[HTML]{bbb6ec}8.233 &\cellcolor[HTML]{b3aeea}9.019 \\
ind\_Latn &\cellcolor[HTML]{e9e7f9}3.410 &\cellcolor[HTML]{e2e0f7}4.049 &\cellcolor[HTML]{e0ddf7}4.338 &\cellcolor[HTML]{dcdaf6}4.714 &\cellcolor[HTML]{d9d6f5}5.080 &\cellcolor[HTML]{d6d3f4}5.402 &\cellcolor[HTML]{d3d0f3}5.643 & &\cellcolor[HTML]{e9e7f9}3.350 &\cellcolor[HTML]{e3e1f8}3.974 &\cellcolor[HTML]{e0def7}4.258 &\cellcolor[HTML]{dddbf6}4.623 &\cellcolor[HTML]{d9d7f5}4.987 &\cellcolor[HTML]{d6d4f4}5.305 &\cellcolor[HTML]{d4d1f4}5.540 \\
isl\_Latn &\cellcolor[HTML]{f1f0fb}2.523 &\cellcolor[HTML]{eae8fa}3.264 &\cellcolor[HTML]{e8e6f9}3.514 &\cellcolor[HTML]{e5e3f8}3.795 &\cellcolor[HTML]{e2e0f7}4.107 &\cellcolor[HTML]{dfdcf6}4.457 &\cellcolor[HTML]{dbd9f5}4.800 & &\cellcolor[HTML]{f1f0fc}2.477 &\cellcolor[HTML]{eae9fa}3.215 &\cellcolor[HTML]{e8e6f9}3.462 &\cellcolor[HTML]{e5e4f8}3.735 &\cellcolor[HTML]{e2e0f7}4.049 &\cellcolor[HTML]{dfddf7}4.395 &\cellcolor[HTML]{dcd9f6}4.729 \\
ita\_Latn &\cellcolor[HTML]{e7e6f9}3.533 &\cellcolor[HTML]{e4e2f8}3.910 &\cellcolor[HTML]{e2e0f7}4.094 &\cellcolor[HTML]{e0ddf7}4.337 &\cellcolor[HTML]{dddbf6}4.613 &\cellcolor[HTML]{dad8f5}4.887 &\cellcolor[HTML]{d8d5f5}5.116 & &\cellcolor[HTML]{e8e6f9}3.502 &\cellcolor[HTML]{e4e2f8}3.868 &\cellcolor[HTML]{e2e0f7}4.047 &\cellcolor[HTML]{e0def7}4.283 &\cellcolor[HTML]{dedbf6}4.552 &\cellcolor[HTML]{dbd8f5}4.821 &\cellcolor[HTML]{d9d6f5}5.043 \\
jpn\_Jpan &\cellcolor[HTML]{e2e0f7}4.071 &\cellcolor[HTML]{dfddf6}4.426 &\cellcolor[HTML]{dddaf6}4.639 &\cellcolor[HTML]{dad7f5}4.950 &\cellcolor[HTML]{d6d3f4}5.361 &\cellcolor[HTML]{d1cef3}5.866 &\cellcolor[HTML]{cbc8f1}6.476 & &\cellcolor[HTML]{e0def7}4.313 &\cellcolor[HTML]{dcdaf6}4.687 &\cellcolor[HTML]{dbd8f5}4.874 &\cellcolor[HTML]{d8d5f5}5.144 &\cellcolor[HTML]{d5d2f4}5.480 &\cellcolor[HTML]{d1cdf3}5.910 &\cellcolor[HTML]{ccc8f1}6.447 \\
kan\_Knda &\cellcolor[HTML]{fafafe}1.560 &\cellcolor[HTML]{d7d4f4}5.260 &\cellcolor[HTML]{d5d2f4}5.488 &\cellcolor[HTML]{d3d0f3}5.615 &\cellcolor[HTML]{d3cff3}5.709 &\cellcolor[HTML]{d2cff3}5.734 &\cellcolor[HTML]{d2cff3}5.743 & &\cellcolor[HTML]{f6f6fd}1.964 &\cellcolor[HTML]{d7d4f4}5.231 &\cellcolor[HTML]{d5d2f4}5.453 &\cellcolor[HTML]{d4d1f3}5.576 &\cellcolor[HTML]{d3d0f3}5.667 &\cellcolor[HTML]{d3d0f3}5.691 &\cellcolor[HTML]{d3cff3}5.706 \\
kat\_Geor &\cellcolor[HTML]{fbfbfe}1.470 &\cellcolor[HTML]{c8c4f0}6.859 &\cellcolor[HTML]{bdb8ed}7.975 &\cellcolor[HTML]{b0aae9}9.356 &\cellcolor[HTML]{a29be6}10.827 &\cellcolor[HTML]{938ce2}12.350 &\cellcolor[HTML]{847cdd}13.899 & &\cellcolor[HTML]{efeefb}2.718 &\cellcolor[HTML]{c7c3f0}6.926 &\cellcolor[HTML]{bdb8ed}7.989 &\cellcolor[HTML]{b1abea}9.267 &\cellcolor[HTML]{a39de6}10.693 &\cellcolor[HTML]{958de2}12.215 &\cellcolor[HTML]{867ede}13.708 \\
kaz\_Cyrl &\cellcolor[HTML]{edebfa}2.983 &\cellcolor[HTML]{d4d1f3}5.550 &\cellcolor[HTML]{cdcaf2}6.254 &\cellcolor[HTML]{c6c2f0}7.026 &\cellcolor[HTML]{bebaed}7.825 &\cellcolor[HTML]{b6b1eb}8.674 &\cellcolor[HTML]{afa9e9}9.492 & &\cellcolor[HTML]{ecebfa}3.007 &\cellcolor[HTML]{d6d3f4}5.371 &\cellcolor[HTML]{cfccf2}6.059 &\cellcolor[HTML]{c8c4f0}6.841 &\cellcolor[HTML]{c0bbee}7.664 &\cellcolor[HTML]{b8b3ec}8.504 &\cellcolor[HTML]{b0abea}9.314 \\
khk\_Cyrl &\cellcolor[HTML]{eeecfb}2.869 &\cellcolor[HTML]{d5d2f4}5.445 &\cellcolor[HTML]{cecbf2}6.159 &\cellcolor[HTML]{c6c2f0}7.007 &\cellcolor[HTML]{beb9ed}7.870 &\cellcolor[HTML]{b6b1eb}8.690 &\cellcolor[HTML]{afa9e9}9.423 & &\cellcolor[HTML]{edebfa}2.962 &\cellcolor[HTML]{d7d4f4}5.291 &\cellcolor[HTML]{d0ccf2}5.996 &\cellcolor[HTML]{c8c4f0}6.859 &\cellcolor[HTML]{bfbbee}7.723 &\cellcolor[HTML]{b8b2ec}8.541 &\cellcolor[HTML]{b1abea}9.268 \\
khm\_Khmr &\cellcolor[HTML]{f8f7fd}1.837 &\cellcolor[HTML]{d9d7f5}5.003 &\cellcolor[HTML]{d6d3f4}5.323 &\cellcolor[HTML]{d4d1f4}5.541 &\cellcolor[HTML]{d3d0f3}5.676 &\cellcolor[HTML]{d2cff3}5.752 &\cellcolor[HTML]{d2cff3}5.781 & &\cellcolor[HTML]{f2f1fc}2.435 &\cellcolor[HTML]{d9d7f5}5.001 &\cellcolor[HTML]{d6d4f4}5.305 &\cellcolor[HTML]{d4d1f4}5.517 &\cellcolor[HTML]{d3d0f3}5.649 &\cellcolor[HTML]{d2cff3}5.720 &\cellcolor[HTML]{d2cff3}5.748 \\
kir\_Cyrl &\cellcolor[HTML]{ebe9fa}3.175 &\cellcolor[HTML]{d4d1f3}5.554 &\cellcolor[HTML]{cecbf2}6.183 &\cellcolor[HTML]{c7c3f0}6.950 &\cellcolor[HTML]{bfbbee}7.727 &\cellcolor[HTML]{b7b2ec}8.552 &\cellcolor[HTML]{b0aae9}9.344 & &\cellcolor[HTML]{eae8f9}3.292 &\cellcolor[HTML]{d6d3f4}5.344 &\cellcolor[HTML]{d0cdf2}5.976 &\cellcolor[HTML]{c9c5f0}6.764 &\cellcolor[HTML]{c1bcee}7.563 &\cellcolor[HTML]{b9b4ec}8.393 &\cellcolor[HTML]{b2acea}9.174 \\
kor\_Hang &\cellcolor[HTML]{e4e2f8}3.902 &\cellcolor[HTML]{e1dff7}4.207 &\cellcolor[HTML]{dfddf6}4.413 &\cellcolor[HTML]{dcdaf6}4.699 &\cellcolor[HTML]{d9d6f5}5.070 &\cellcolor[HTML]{d5d2f4}5.487 &\cellcolor[HTML]{d0cdf2}5.936 & &\cellcolor[HTML]{e8e6f9}3.505 &\cellcolor[HTML]{e3e1f8}4.028 &\cellcolor[HTML]{e0def7}4.282 &\cellcolor[HTML]{dddbf6}4.606 &\cellcolor[HTML]{dad7f5}4.982 &\cellcolor[HTML]{d6d3f4}5.395 &\cellcolor[HTML]{d1cef3}5.838 \\
lit\_Latn &\cellcolor[HTML]{f2f1fc}2.389 &\cellcolor[HTML]{eae8f9}3.299 &\cellcolor[HTML]{e7e5f9}3.557 &\cellcolor[HTML]{e4e2f8}3.865 &\cellcolor[HTML]{e1def7}4.236 &\cellcolor[HTML]{dddaf6}4.657 &\cellcolor[HTML]{d9d6f5}5.088 & &\cellcolor[HTML]{f1f0fc}2.469 &\cellcolor[HTML]{eae9fa}3.233 &\cellcolor[HTML]{e8e6f9}3.496 &\cellcolor[HTML]{e5e3f8}3.805 &\cellcolor[HTML]{e1dff7}4.172 &\cellcolor[HTML]{dddbf6}4.584 &\cellcolor[HTML]{d9d7f5}5.008 \\
lvs\_Latn &\cellcolor[HTML]{f3f2fc}2.341 &\cellcolor[HTML]{eae8f9}3.283 &\cellcolor[HTML]{e7e5f9}3.562 &\cellcolor[HTML]{e4e2f8}3.896 &\cellcolor[HTML]{e0def7}4.272 &\cellcolor[HTML]{dcdaf6}4.688 &\cellcolor[HTML]{d8d6f5}5.094 & &\cellcolor[HTML]{f3f2fc}2.331 &\cellcolor[HTML]{eae9fa}3.220 &\cellcolor[HTML]{e8e6f9}3.500 &\cellcolor[HTML]{e5e3f8}3.829 &\cellcolor[HTML]{e1dff7}4.201 &\cellcolor[HTML]{dddbf6}4.609 &\cellcolor[HTML]{d9d7f5}5.008 \\
mal\_Mlym &\cellcolor[HTML]{f9f8fe}1.696 &\cellcolor[HTML]{d6d3f4}5.394 &\cellcolor[HTML]{d3d0f3}5.662 &\cellcolor[HTML]{d1cef3}5.825 &\cellcolor[HTML]{d0cdf2}5.930 &\cellcolor[HTML]{d0cdf2}5.969 &\cellcolor[HTML]{d0cdf2}5.984 & &\cellcolor[HTML]{f5f5fd}2.063 &\cellcolor[HTML]{d6d3f4}5.365 &\cellcolor[HTML]{d3d0f3}5.623 &\cellcolor[HTML]{d2cff3}5.777 &\cellcolor[HTML]{d1cef3}5.894 &\cellcolor[HTML]{d0cdf2}5.931 &\cellcolor[HTML]{d0cdf2}5.945 \\
mar\_Deva &\cellcolor[HTML]{dcdaf6}4.677 &\cellcolor[HTML]{d7d4f4}5.265 &\cellcolor[HTML]{d6d3f4}5.348 &\cellcolor[HTML]{d5d2f4}5.414 &\cellcolor[HTML]{d5d2f4}5.461 &\cellcolor[HTML]{d5d2f4}5.482 &\cellcolor[HTML]{d5d2f4}5.491 & &\cellcolor[HTML]{eeedfb}2.789 &\cellcolor[HTML]{d9d6f5}5.067 &\cellcolor[HTML]{d7d4f4}5.260 &\cellcolor[HTML]{d6d3f4}5.360 &\cellcolor[HTML]{d5d2f4}5.416 &\cellcolor[HTML]{d5d2f4}5.441 &\cellcolor[HTML]{d5d2f4}5.453 \\
mkd\_Cyrl &\cellcolor[HTML]{e0ddf7}4.349 &\cellcolor[HTML]{d2cff3}5.733 &\cellcolor[HTML]{cfcbf2}6.113 &\cellcolor[HTML]{cac6f1}6.591 &\cellcolor[HTML]{c4c0ef}7.195 &\cellcolor[HTML]{bebaed}7.851 &\cellcolor[HTML]{b8b3ec}8.481 & &\cellcolor[HTML]{e3e1f8}4.005 &\cellcolor[HTML]{d5d2f4}5.428 &\cellcolor[HTML]{d1cef3}5.837 &\cellcolor[HTML]{ccc9f1}6.371 &\cellcolor[HTML]{c6c2f0}7.012 &\cellcolor[HTML]{c0bbee}7.686 &\cellcolor[HTML]{bab5ec}8.333 \\
mya\_Mymr &\cellcolor[HTML]{fbfbfe}1.474 &\cellcolor[HTML]{d6d4f4}5.304 &\cellcolor[HTML]{d5d2f4}5.497 &\cellcolor[HTML]{d4d1f3}5.591 &\cellcolor[HTML]{d3d0f3}5.635 &\cellcolor[HTML]{d3d0f3}5.651 &\cellcolor[HTML]{d3d0f3}5.661 & &\cellcolor[HTML]{f7f6fd}1.888 &\cellcolor[HTML]{d6d3f4}5.309 &\cellcolor[HTML]{d5d2f4}5.486 &\cellcolor[HTML]{d4d1f3}5.571 &\cellcolor[HTML]{d4d0f3}5.611 &\cellcolor[HTML]{d3d0f3}5.627 &\cellcolor[HTML]{d3d0f3}5.635 \\
nld\_Latn &\cellcolor[HTML]{e9e7f9}3.402 &\cellcolor[HTML]{e4e3f8}3.835 &\cellcolor[HTML]{e3e0f7}4.038 &\cellcolor[HTML]{e0def7}4.295 &\cellcolor[HTML]{dddbf6}4.574 &\cellcolor[HTML]{dbd8f5}4.828 &\cellcolor[HTML]{d9d6f5}5.044 & &\cellcolor[HTML]{e9e7f9}3.364 &\cellcolor[HTML]{e5e3f8}3.784 &\cellcolor[HTML]{e3e1f8}3.983 &\cellcolor[HTML]{e1def7}4.236 &\cellcolor[HTML]{dedcf6}4.508 &\cellcolor[HTML]{dcd9f6}4.758 &\cellcolor[HTML]{dad7f5}4.969 \\
nno\_Latn &\cellcolor[HTML]{eceafa}3.079 &\cellcolor[HTML]{e7e5f9}3.595 &\cellcolor[HTML]{e5e3f8}3.768 &\cellcolor[HTML]{e3e1f8}3.977 &\cellcolor[HTML]{e1dff7}4.221 &\cellcolor[HTML]{dedcf6}4.465 &\cellcolor[HTML]{dcdaf6}4.688 & &\cellcolor[HTML]{ecebfa}3.031 &\cellcolor[HTML]{e7e6f9}3.538 &\cellcolor[HTML]{e6e4f8}3.712 &\cellcolor[HTML]{e4e2f8}3.916 &\cellcolor[HTML]{e1dff7}4.156 &\cellcolor[HTML]{dfddf7}4.398 &\cellcolor[HTML]{dddbf6}4.615 \\
nob\_Latn &\cellcolor[HTML]{ebe9fa}3.196 &\cellcolor[HTML]{e7e5f9}3.600 &\cellcolor[HTML]{e5e3f8}3.763 &\cellcolor[HTML]{e3e1f8}3.980 &\cellcolor[HTML]{e1dff7}4.222 &\cellcolor[HTML]{dfdcf6}4.460 &\cellcolor[HTML]{dcdaf6}4.689 & &\cellcolor[HTML]{ebeafa}3.149 &\cellcolor[HTML]{e7e6f9}3.545 &\cellcolor[HTML]{e6e4f8}3.705 &\cellcolor[HTML]{e4e2f8}3.915 &\cellcolor[HTML]{e1dff7}4.153 &\cellcolor[HTML]{dfddf7}4.389 &\cellcolor[HTML]{dddbf6}4.613 \\
npi\_Deva &\cellcolor[HTML]{dbd8f5}4.837 &\cellcolor[HTML]{d6d3f4}5.385 &\cellcolor[HTML]{d5d2f4}5.483 &\cellcolor[HTML]{d4d1f3}5.572 &\cellcolor[HTML]{d3d0f3}5.626 &\cellcolor[HTML]{d3d0f3}5.653 &\cellcolor[HTML]{d3d0f3}5.668 & &\cellcolor[HTML]{eeedfb}2.784 &\cellcolor[HTML]{d8d5f4}5.175 &\cellcolor[HTML]{d6d3f4}5.371 &\cellcolor[HTML]{d5d2f4}5.503 &\cellcolor[HTML]{d4d1f3}5.574 &\cellcolor[HTML]{d4d0f3}5.604 &\cellcolor[HTML]{d3d0f3}5.626 \\
ory\_Orya &1.094 &\cellcolor[HTML]{d8d5f4}5.192 &\cellcolor[HTML]{d6d3f4}5.401 &\cellcolor[HTML]{d5d1f4}5.506 &\cellcolor[HTML]{d4d1f3}5.567 &\cellcolor[HTML]{d4d1f3}5.591 &\cellcolor[HTML]{d3d0f3}5.614 & &\cellcolor[HTML]{fcfcff}1.349 &\cellcolor[HTML]{d8d5f4}5.182 &\cellcolor[HTML]{d6d3f4}5.392 &\cellcolor[HTML]{d5d2f4}5.497 &\cellcolor[HTML]{d4d1f3}5.554 &\cellcolor[HTML]{d4d1f3}5.579 &\cellcolor[HTML]{d4d1f3}5.601 \\
pan\_Guru &\cellcolor[HTML]{faf9fe}1.621 &\cellcolor[HTML]{dcd9f6}4.748 &\cellcolor[HTML]{dad7f5}4.922 &\cellcolor[HTML]{d9d7f5}5.003 &\cellcolor[HTML]{d9d6f5}5.042 &\cellcolor[HTML]{d9d6f5}5.058 &\cellcolor[HTML]{d9d6f5}5.064 & &\cellcolor[HTML]{f9f8fe}1.712 &\cellcolor[HTML]{dbd9f5}4.796 &\cellcolor[HTML]{dad7f5}4.970 &\cellcolor[HTML]{d9d6f5}5.055 &\cellcolor[HTML]{d8d6f5}5.096 &\cellcolor[HTML]{d8d6f5}5.112 &\cellcolor[HTML]{d8d5f5}5.119 \\
pol\_Latn &\cellcolor[HTML]{eeecfb}2.886 &\cellcolor[HTML]{e9e8f9}3.342 &\cellcolor[HTML]{e7e5f9}3.579 &\cellcolor[HTML]{e4e2f8}3.886 &\cellcolor[HTML]{e1def7}4.236 &\cellcolor[HTML]{dddaf6}4.628 &\cellcolor[HTML]{d9d6f5}5.057 & &\cellcolor[HTML]{ecebfa}3.041 &\cellcolor[HTML]{e8e7f9}3.412 &\cellcolor[HTML]{e6e5f9}3.623 &\cellcolor[HTML]{e4e2f8}3.889 &\cellcolor[HTML]{e1dff7}4.213 &\cellcolor[HTML]{dddbf6}4.582 &\cellcolor[HTML]{d9d7f5}4.997 \\
por\_Latn &\cellcolor[HTML]{e6e4f8}3.686 &\cellcolor[HTML]{e3e1f8}3.951 &\cellcolor[HTML]{e2e0f7}4.105 &\cellcolor[HTML]{e0def7}4.308 &\cellcolor[HTML]{dedbf6}4.547 &\cellcolor[HTML]{dbd9f5}4.804 &\cellcolor[HTML]{d9d6f5}5.017 & &\cellcolor[HTML]{e6e4f8}3.682 &\cellcolor[HTML]{e4e2f8}3.926 &\cellcolor[HTML]{e2e0f7}4.073 &\cellcolor[HTML]{e0def7}4.260 &\cellcolor[HTML]{dedcf6}4.495 &\cellcolor[HTML]{dcd9f6}4.741 &\cellcolor[HTML]{dad7f5}4.947 \\
ron\_Latn &\cellcolor[HTML]{eceafa}3.092 &\cellcolor[HTML]{e5e4f8}3.728 &\cellcolor[HTML]{e4e2f8}3.918 &\cellcolor[HTML]{e1dff7}4.166 &\cellcolor[HTML]{dfdcf6}4.435 &\cellcolor[HTML]{dcd9f6}4.727 &\cellcolor[HTML]{d9d7f5}4.996 & &\cellcolor[HTML]{ecebfa}3.056 &\cellcolor[HTML]{e6e4f8}3.679 &\cellcolor[HTML]{e4e2f8}3.867 &\cellcolor[HTML]{e2e0f7}4.113 &\cellcolor[HTML]{dfddf7}4.377 &\cellcolor[HTML]{dddaf6}4.662 &\cellcolor[HTML]{dad7f5}4.924 \\
rus\_Cyrl &\cellcolor[HTML]{d1cdf3}5.928 &\cellcolor[HTML]{cdcaf2}6.255 &\cellcolor[HTML]{ccc8f1}6.435 &\cellcolor[HTML]{c9c5f0}6.720 &\cellcolor[HTML]{c5c1ef}7.164 &\cellcolor[HTML]{bfbbee}7.750 &\cellcolor[HTML]{b9b4ec}8.413 & &\cellcolor[HTML]{d6d3f4}5.371 &\cellcolor[HTML]{d1cef3}5.825 &\cellcolor[HTML]{cfccf2}6.089 &\cellcolor[HTML]{ccc8f1}6.433 &\cellcolor[HTML]{c7c3f0}6.946 &\cellcolor[HTML]{c1bcee}7.561 &\cellcolor[HTML]{bab5ec}8.253 \\
sin\_Sinh &\cellcolor[HTML]{fbfbfe}1.476 &\cellcolor[HTML]{d9d6f5}5.029 &\cellcolor[HTML]{d7d4f4}5.268 &\cellcolor[HTML]{d5d2f4}5.422 &\cellcolor[HTML]{d5d2f4}5.494 &\cellcolor[HTML]{d4d1f4}5.524 &\cellcolor[HTML]{d4d1f4}5.540 & &\cellcolor[HTML]{f8f7fd}1.822 &\cellcolor[HTML]{dad8f5}4.907 &\cellcolor[HTML]{d8d5f5}5.131 &\cellcolor[HTML]{d7d4f4}5.273 &\cellcolor[HTML]{d6d3f4}5.340 &\cellcolor[HTML]{d6d3f4}5.369 &\cellcolor[HTML]{d6d3f4}5.383 \\
slk\_Latn &\cellcolor[HTML]{eeecfb}2.871 &\cellcolor[HTML]{e8e6f9}3.470 &\cellcolor[HTML]{e6e4f8}3.667 &\cellcolor[HTML]{e4e2f8}3.912 &\cellcolor[HTML]{e1dff7}4.215 &\cellcolor[HTML]{dddbf6}4.571 &\cellcolor[HTML]{dad7f5}4.961 & &\cellcolor[HTML]{f1f0fc}2.510 &\cellcolor[HTML]{ebe9fa}3.184 &\cellcolor[HTML]{e8e7f9}3.443 &\cellcolor[HTML]{e5e3f8}3.751 &\cellcolor[HTML]{e2e0f7}4.076 &\cellcolor[HTML]{dedcf6}4.470 &\cellcolor[HTML]{dbd8f5}4.859 \\
slv\_Latn &\cellcolor[HTML]{efeefb}2.744 &\cellcolor[HTML]{eae8fa}3.259 &\cellcolor[HTML]{e8e7f9}3.447 &\cellcolor[HTML]{e6e4f8}3.679 &\cellcolor[HTML]{e3e1f8}3.960 &\cellcolor[HTML]{e0def7}4.262 &\cellcolor[HTML]{dddbf6}4.592 & &\cellcolor[HTML]{f0effb}2.599 &\cellcolor[HTML]{ebeafa}3.147 &\cellcolor[HTML]{e9e7f9}3.353 &\cellcolor[HTML]{e7e5f9}3.594 &\cellcolor[HTML]{e4e2f8}3.884 &\cellcolor[HTML]{e1dff7}4.185 &\cellcolor[HTML]{dedcf6}4.514 \\
spa\_Latn &\cellcolor[HTML]{e5e3f8}3.821 &\cellcolor[HTML]{e2e0f7}4.055 &\cellcolor[HTML]{e1dff7}4.195 &\cellcolor[HTML]{dfddf7}4.376 &\cellcolor[HTML]{dddbf6}4.598 &\cellcolor[HTML]{dbd8f5}4.824 &\cellcolor[HTML]{d9d7f5}5.008 & &\cellcolor[HTML]{e5e3f8}3.806 &\cellcolor[HTML]{e3e1f8}4.030 &\cellcolor[HTML]{e1dff7}4.161 &\cellcolor[HTML]{e0def7}4.329 &\cellcolor[HTML]{dedbf6}4.545 &\cellcolor[HTML]{dcd9f6}4.766 &\cellcolor[HTML]{dad7f5}4.942 \\
srp\_Cyrl &\cellcolor[HTML]{e3e1f8}4.013 &\cellcolor[HTML]{d6d3f4}5.356 &\cellcolor[HTML]{d3cff3}5.717 &\cellcolor[HTML]{cecbf2}6.151 &\cellcolor[HTML]{c9c6f1}6.668 &\cellcolor[HTML]{c4bfef}7.278 &\cellcolor[HTML]{beb9ed}7.919 & &\cellcolor[HTML]{e6e4f8}3.694 &\cellcolor[HTML]{d9d6f5}5.050 &\cellcolor[HTML]{d5d2f4}5.455 &\cellcolor[HTML]{d1cdf3}5.925 &\cellcolor[HTML]{cbc7f1}6.492 &\cellcolor[HTML]{c5c1ef}7.130 &\cellcolor[HTML]{bfbaee}7.781 \\
swe\_Latn &\cellcolor[HTML]{eae9fa}3.223 &\cellcolor[HTML]{e7e5f9}3.607 &\cellcolor[HTML]{e5e3f8}3.785 &\cellcolor[HTML]{e3e1f8}4.023 &\cellcolor[HTML]{e0def7}4.299 &\cellcolor[HTML]{dddbf6}4.609 &\cellcolor[HTML]{dad8f5}4.883 & &\cellcolor[HTML]{ebe9fa}3.177 &\cellcolor[HTML]{e7e5f9}3.556 &\cellcolor[HTML]{e5e4f8}3.727 &\cellcolor[HTML]{e3e1f8}3.962 &\cellcolor[HTML]{e1dff7}4.231 &\cellcolor[HTML]{dedbf6}4.538 &\cellcolor[HTML]{dbd9f5}4.804 \\
swh\_Latn &\cellcolor[HTML]{f0effb}2.636 &\cellcolor[HTML]{e8e6f9}3.495 &\cellcolor[HTML]{e5e3f8}3.797 &\cellcolor[HTML]{e2e0f7}4.130 &\cellcolor[HTML]{dedcf6}4.480 &\cellcolor[HTML]{dcd9f6}4.771 &\cellcolor[HTML]{d9d7f5}4.997 & &\cellcolor[HTML]{f0effb}2.586 &\cellcolor[HTML]{e8e7f9}3.442 &\cellcolor[HTML]{e5e4f8}3.734 &\cellcolor[HTML]{e2e0f7}4.067 &\cellcolor[HTML]{dfddf7}4.409 &\cellcolor[HTML]{dcdaf6}4.695 &\cellcolor[HTML]{dad7f5}4.917 \\
tam\_Taml &\cellcolor[HTML]{f6f5fd}2.029 &\cellcolor[HTML]{d4d1f3}5.590 &\cellcolor[HTML]{d2cff3}5.723 &\cellcolor[HTML]{d2cff3}5.785 &\cellcolor[HTML]{d2cef3}5.806 &\cellcolor[HTML]{d2cef3}5.811 &\cellcolor[HTML]{d2cef3}5.815 & &\cellcolor[HTML]{f1f0fc}2.497 &\cellcolor[HTML]{d4d1f3}5.565 &\cellcolor[HTML]{d3d0f3}5.693 &\cellcolor[HTML]{d2cff3}5.751 &\cellcolor[HTML]{d2cff3}5.771 &\cellcolor[HTML]{d2cff3}5.776 &\cellcolor[HTML]{d2cff3}5.781 \\
tel\_Telu &\cellcolor[HTML]{fafafe}1.570 &\cellcolor[HTML]{d9d6f5}5.017 &\cellcolor[HTML]{d8d5f5}5.163 &\cellcolor[HTML]{d7d4f4}5.238 &\cellcolor[HTML]{d7d4f4}5.288 &\cellcolor[HTML]{d6d4f4}5.303 &\cellcolor[HTML]{d6d3f4}5.309 & &\cellcolor[HTML]{f8f7fd}1.831 &\cellcolor[HTML]{d9d7f5}4.989 &\cellcolor[HTML]{d8d5f5}5.133 &\cellcolor[HTML]{d7d5f4}5.205 &\cellcolor[HTML]{d7d4f4}5.253 &\cellcolor[HTML]{d7d4f4}5.268 &\cellcolor[HTML]{d7d4f4}5.274 \\
tha\_Thai &\cellcolor[HTML]{cecbf2}6.146 &\cellcolor[HTML]{c9c5f0}6.726 &\cellcolor[HTML]{c6c2ef}7.068 &\cellcolor[HTML]{c1bdee}7.543 &\cellcolor[HTML]{bcb8ed}8.048 &\cellcolor[HTML]{b7b2ec}8.551 &\cellcolor[HTML]{b3adea}9.042 & &\cellcolor[HTML]{d7d5f4}5.197 &\cellcolor[HTML]{cbc7f1}6.494 &\cellcolor[HTML]{c7c3f0}6.913 &\cellcolor[HTML]{c2beef}7.414 &\cellcolor[HTML]{bdb9ed}7.948 &\cellcolor[HTML]{b8b3ec}8.460 &\cellcolor[HTML]{b4aeeb}8.953 \\
tur\_Latn &\cellcolor[HTML]{e4e2f8}3.882 &\cellcolor[HTML]{e1dff7}4.194 &\cellcolor[HTML]{e0ddf7}4.332 &\cellcolor[HTML]{dedbf6}4.551 &\cellcolor[HTML]{dbd8f5}4.848 &\cellcolor[HTML]{d8d5f4}5.189 &\cellcolor[HTML]{d4d1f3}5.554 & &\cellcolor[HTML]{eae8f9}3.298 &\cellcolor[HTML]{e5e3f8}3.809 &\cellcolor[HTML]{e2e0f7}4.041 &\cellcolor[HTML]{e0ddf7}4.338 &\cellcolor[HTML]{dcdaf6}4.683 &\cellcolor[HTML]{d9d6f5}5.054 &\cellcolor[HTML]{d5d2f4}5.437 \\
ukr\_Cyrl &\cellcolor[HTML]{d4d1f4}5.512 &\cellcolor[HTML]{d1cdf3}5.910 &\cellcolor[HTML]{cfcbf2}6.137 &\cellcolor[HTML]{cbc7f1}6.487 &\cellcolor[HTML]{c7c3f0}6.955 &\cellcolor[HTML]{c1bdee}7.554 &\cellcolor[HTML]{bbb6ec}8.206 & &\cellcolor[HTML]{e7e5f9}3.564 &\cellcolor[HTML]{d9d6f5}5.024 &\cellcolor[HTML]{d5d2f4}5.445 &\cellcolor[HTML]{d0cdf2}5.965 &\cellcolor[HTML]{cac6f1}6.594 &\cellcolor[HTML]{c4bfef}7.282 &\cellcolor[HTML]{bdb8ed}7.998 \\
urd\_Arab &\cellcolor[HTML]{efedfb}2.774 &\cellcolor[HTML]{d7d4f4}5.256 &\cellcolor[HTML]{d3d0f3}5.621 &\cellcolor[HTML]{cfccf2}6.048 &\cellcolor[HTML]{cbc8f1}6.463 &\cellcolor[HTML]{c8c4f0}6.813 &\cellcolor[HTML]{c6c2f0}7.065 & &\cellcolor[HTML]{f0effb}2.664 &\cellcolor[HTML]{dbd9f6}4.779 &\cellcolor[HTML]{d7d4f4}5.251 &\cellcolor[HTML]{d2cef3}5.807 &\cellcolor[HTML]{cdc9f2}6.294 &\cellcolor[HTML]{c9c5f0}6.691 &\cellcolor[HTML]{c7c3f0}6.963 \\
uzn\_Latn &\cellcolor[HTML]{f0effb}2.595 &\cellcolor[HTML]{e8e7f9}3.424 &\cellcolor[HTML]{e6e4f8}3.719 &\cellcolor[HTML]{e2e0f7}4.049 &\cellcolor[HTML]{dfddf6}4.417 &\cellcolor[HTML]{dbd9f5}4.803 &\cellcolor[HTML]{d8d5f5}5.132 & &\cellcolor[HTML]{f1f0fb}2.522 &\cellcolor[HTML]{e9e8f9}3.346 &\cellcolor[HTML]{e6e4f9}3.644 &\cellcolor[HTML]{e3e1f8}3.973 &\cellcolor[HTML]{e0ddf7}4.345 &\cellcolor[HTML]{dcd9f6}4.725 &\cellcolor[HTML]{d9d6f5}5.049 \\
vie\_Latn &\cellcolor[HTML]{dbd8f5}4.856 &\cellcolor[HTML]{d9d6f5}5.076 &\cellcolor[HTML]{d8d5f4}5.175 &\cellcolor[HTML]{d7d4f4}5.244 &\cellcolor[HTML]{d7d4f4}5.283 &\cellcolor[HTML]{d6d4f4}5.302 &\cellcolor[HTML]{d6d3f4}5.313 & &\cellcolor[HTML]{dddaf6}4.631 &\cellcolor[HTML]{d9d7f5}5.008 &\cellcolor[HTML]{d8d6f5}5.110 &\cellcolor[HTML]{d8d5f4}5.177 &\cellcolor[HTML]{d7d4f4}5.215 &\cellcolor[HTML]{d7d4f4}5.234 &\cellcolor[HTML]{d7d4f4}5.245 \\
zsm\_Latn &\cellcolor[HTML]{e9e7f9}3.357 &\cellcolor[HTML]{e1dff7}4.163 &\cellcolor[HTML]{dedcf6}4.514 &\cellcolor[HTML]{dad8f5}4.906 &\cellcolor[HTML]{d6d3f4}5.311 &\cellcolor[HTML]{d3d0f3}5.627 &\cellcolor[HTML]{d2cef3}5.815 & &\cellcolor[HTML]{eae8f9}3.301 &\cellcolor[HTML]{e2e0f7}4.091 &\cellcolor[HTML]{dfdcf6}4.431 &\cellcolor[HTML]{dbd9f5}4.815 &\cellcolor[HTML]{d7d4f4}5.214 &\cellcolor[HTML]{d4d1f4}5.523 &\cellcolor[HTML]{d3cff3}5.710 \\
\bottomrule
\end{tabular}

\begin{tabular}{lcccccccccccccccc}\toprule
&\multicolumn{7}{c}{\textbf{Mistral Nemo}} & &\multicolumn{7}{c}{\textbf{Llama-2}} \\\cmidrule{2-8}\cmidrule{10-16}
\textbf{Lang.} &\textbf{0} &\textbf{1000} &\textbf{2000} &\textbf{4000} &\textbf{8000} &\textbf{16000} &\textbf{32000} & &\textbf{0} &\textbf{1000} &\textbf{2000} &\textbf{4000} &\textbf{8000} &\textbf{16000} &\textbf{32000} \\\midrule
afr\_Latn &\cellcolor[HTML]{e9e8f9}3.343 &\cellcolor[HTML]{e5e4f8}3.728 &\cellcolor[HTML]{e4e2f8}3.886 &\cellcolor[HTML]{e2e0f7}4.087 &\cellcolor[HTML]{e0def7}4.294 &\cellcolor[HTML]{dedcf6}4.499 &\cellcolor[HTML]{dddaf6}4.666 & &\cellcolor[HTML]{edecfa}2.932 &\cellcolor[HTML]{e8e6f9}3.461 &\cellcolor[HTML]{e6e4f8}3.660 &\cellcolor[HTML]{e4e2f8}3.903 &\cellcolor[HTML]{e2dff7}4.145 &\cellcolor[HTML]{dfddf7}4.379 &\cellcolor[HTML]{dddbf6}4.580 \\
als\_Latn &\cellcolor[HTML]{ecebfa}3.049 &\cellcolor[HTML]{e7e5f9}3.613 &\cellcolor[HTML]{e5e3f8}3.817 &\cellcolor[HTML]{e2e0f7}4.061 &\cellcolor[HTML]{e0ddf7}4.332 &\cellcolor[HTML]{dddaf6}4.623 &\cellcolor[HTML]{dad8f5}4.887 & &\cellcolor[HTML]{f3f3fc}2.259 &\cellcolor[HTML]{e9e7f9}3.384 &\cellcolor[HTML]{e7e5f9}3.616 &\cellcolor[HTML]{e4e2f8}3.887 &\cellcolor[HTML]{e1dff7}4.188 &\cellcolor[HTML]{dedcf6}4.503 &\cellcolor[HTML]{dbd9f5}4.797 \\
arb\_Arab &\cellcolor[HTML]{cfcbf2}6.115 &\cellcolor[HTML]{cecaf2}6.232 &\cellcolor[HTML]{cdc9f1}6.333 &\cellcolor[HTML]{cbc7f1}6.501 &\cellcolor[HTML]{c8c4f0}6.786 &\cellcolor[HTML]{c5c0ef}7.182 &\cellcolor[HTML]{c0bcee}7.643 & &\cellcolor[HTML]{f6f5fd}1.988 &\cellcolor[HTML]{e1def7}4.248 &\cellcolor[HTML]{dbd9f6}4.780 &\cellcolor[HTML]{d5d2f4}5.432 &\cellcolor[HTML]{cecbf2}6.144 &\cellcolor[HTML]{c8c4f0}6.845 &\cellcolor[HTML]{c1bdee}7.529 \\
ary\_Arab &\cellcolor[HTML]{d5d2f4}5.411 &\cellcolor[HTML]{d4d1f3}5.600 &\cellcolor[HTML]{d3cff3}5.704 &\cellcolor[HTML]{d1cef3}5.871 &\cellcolor[HTML]{cfcbf2}6.104 &\cellcolor[HTML]{ccc8f1}6.428 &\cellcolor[HTML]{c8c4f0}6.857 & &\cellcolor[HTML]{f6f5fd}1.982 &\cellcolor[HTML]{e3e1f8}3.996 &\cellcolor[HTML]{dedcf6}4.479 &\cellcolor[HTML]{dad7f5}4.958 &\cellcolor[HTML]{d4d1f4}5.549 &\cellcolor[HTML]{cecbf2}6.139 &\cellcolor[HTML]{c9c5f0}6.702 \\
azj\_Latn &\cellcolor[HTML]{e9e8f9}3.311 &\cellcolor[HTML]{e4e2f8}3.845 &\cellcolor[HTML]{e2e0f7}4.133 &\cellcolor[HTML]{dedcf6}4.511 &\cellcolor[HTML]{dad8f5}4.916 &\cellcolor[HTML]{d6d3f4}5.334 &\cellcolor[HTML]{d2cff3}5.765 & &\cellcolor[HTML]{f5f5fd}2.057 &\cellcolor[HTML]{e9e7f9}3.387 &\cellcolor[HTML]{e5e3f8}3.776 &\cellcolor[HTML]{e0def7}4.251 &\cellcolor[HTML]{dcd9f6}4.742 &\cellcolor[HTML]{d7d5f4}5.210 &\cellcolor[HTML]{d3d0f3}5.658 \\
bel\_Cyrl &\cellcolor[HTML]{dedcf6}4.522 &\cellcolor[HTML]{d6d3f4}5.344 &\cellcolor[HTML]{d2cff3}5.744 &\cellcolor[HTML]{cecaf2}6.241 &\cellcolor[HTML]{c8c4f0}6.850 &\cellcolor[HTML]{c1bdee}7.530 &\cellcolor[HTML]{bbb6ec}8.214 & &\cellcolor[HTML]{e6e4f8}3.685 &\cellcolor[HTML]{d9d6f5}5.032 &\cellcolor[HTML]{d5d2f4}5.498 &\cellcolor[HTML]{cfccf2}6.068 &\cellcolor[HTML]{c9c5f0}6.746 &\cellcolor[HTML]{c2bdee}7.467 &\cellcolor[HTML]{bbb6ed}8.154 \\
ben\_Beng &\cellcolor[HTML]{cfccf2}6.038 &\cellcolor[HTML]{c6c2f0}6.980 &\cellcolor[HTML]{c1bcee}7.599 &\cellcolor[HTML]{b9b4ec}8.421 &\cellcolor[HTML]{b0aae9}9.345 &\cellcolor[HTML]{a8a1e7}10.226 &\cellcolor[HTML]{a19ae5}10.947 & &\cellcolor[HTML]{f5f5fd}2.070 &\cellcolor[HTML]{cecbf2}6.151 &\cellcolor[HTML]{c6c2f0}7.011 &\cellcolor[HTML]{bdb8ed}7.987 &\cellcolor[HTML]{b3aeea}8.977 &\cellcolor[HTML]{aba5e8}9.867 &\cellcolor[HTML]{a59ee6}10.530 \\
bos\_Latn &\cellcolor[HTML]{eae9fa}3.210 &\cellcolor[HTML]{e8e6f9}3.462 &\cellcolor[HTML]{e7e5f9}3.608 &\cellcolor[HTML]{e5e3f8}3.801 &\cellcolor[HTML]{e3e0f7}4.036 &\cellcolor[HTML]{e0def7}4.329 &\cellcolor[HTML]{dddaf6}4.633 & &\cellcolor[HTML]{f0effb}2.597 &\cellcolor[HTML]{ebeafa}3.099 &\cellcolor[HTML]{e9e8f9}3.318 &\cellcolor[HTML]{e7e5f9}3.578 &\cellcolor[HTML]{e4e2f8}3.867 &\cellcolor[HTML]{e1dff7}4.199 &\cellcolor[HTML]{dedbf6}4.541 \\
bul\_Cyrl &\cellcolor[HTML]{d5d2f4}5.482 &\cellcolor[HTML]{d0ccf2}6.032 &\cellcolor[HTML]{cdcaf2}6.283 &\cellcolor[HTML]{cac6f1}6.636 &\cellcolor[HTML]{c5c1ef}7.090 &\cellcolor[HTML]{c0bcee}7.639 &\cellcolor[HTML]{bbb6ed}8.193 & &\cellcolor[HTML]{dedbf6}4.536 &\cellcolor[HTML]{d5d2f4}5.473 &\cellcolor[HTML]{d1cef3}5.838 &\cellcolor[HTML]{cdc9f2}6.296 &\cellcolor[HTML]{c7c3f0}6.883 &\cellcolor[HTML]{c1bdee}7.520 &\cellcolor[HTML]{bbb7ed}8.139 \\
cat\_Latn &\cellcolor[HTML]{e6e4f8}3.716 &\cellcolor[HTML]{e4e2f8}3.926 &\cellcolor[HTML]{e3e0f7}4.039 &\cellcolor[HTML]{e1dff7}4.189 &\cellcolor[HTML]{e0ddf7}4.353 &\cellcolor[HTML]{dedbf6}4.542 &\cellcolor[HTML]{dcdaf6}4.689 & &\cellcolor[HTML]{ebe9fa}3.175 &\cellcolor[HTML]{e8e6f9}3.505 &\cellcolor[HTML]{e6e4f9}3.653 &\cellcolor[HTML]{e4e2f8}3.848 &\cellcolor[HTML]{e2e0f7}4.047 &\cellcolor[HTML]{e0def7}4.269 &\cellcolor[HTML]{dfdcf6}4.442 \\
ces\_Latn &\cellcolor[HTML]{e9e8f9}3.311 &\cellcolor[HTML]{e7e5f9}3.565 &\cellcolor[HTML]{e5e4f8}3.730 &\cellcolor[HTML]{e3e1f8}3.951 &\cellcolor[HTML]{e1def7}4.233 &\cellcolor[HTML]{dddbf6}4.589 &\cellcolor[HTML]{dad7f5}4.962 & &\cellcolor[HTML]{efeefb}2.710 &\cellcolor[HTML]{ebe9fa}3.184 &\cellcolor[HTML]{e8e7f9}3.418 &\cellcolor[HTML]{e6e4f8}3.709 &\cellcolor[HTML]{e2e0f7}4.047 &\cellcolor[HTML]{dfdcf6}4.442 &\cellcolor[HTML]{dbd8f5}4.850 \\
cmn\_Hani &\cellcolor[HTML]{eceafa}3.094 &\cellcolor[HTML]{e8e7f9}3.448 &\cellcolor[HTML]{e6e5f9}3.631 &\cellcolor[HTML]{e4e2f8}3.849 &\cellcolor[HTML]{e1dff7}4.147 &\cellcolor[HTML]{dedcf6}4.465 &\cellcolor[HTML]{dbd9f5}4.811 & &\cellcolor[HTML]{f6f6fd}1.963 &\cellcolor[HTML]{f0effb}2.635 &\cellcolor[HTML]{efedfb}2.767 &\cellcolor[HTML]{eeedfb}2.822 &\cellcolor[HTML]{eeedfb}2.829 &\cellcolor[HTML]{e4e2f8}3.935 &\cellcolor[HTML]{dfddf7}4.412 \\
dan\_Latn &\cellcolor[HTML]{e9e7f9}3.380 &\cellcolor[HTML]{e6e4f8}3.673 &\cellcolor[HTML]{e5e3f8}3.818 &\cellcolor[HTML]{e3e1f8}4.007 &\cellcolor[HTML]{e1dff7}4.216 &\cellcolor[HTML]{dfdcf6}4.453 &\cellcolor[HTML]{dcdaf6}4.679 & &\cellcolor[HTML]{edecfa}2.900 &\cellcolor[HTML]{e9e7f9}3.379 &\cellcolor[HTML]{e7e5f9}3.570 &\cellcolor[HTML]{e5e3f8}3.798 &\cellcolor[HTML]{e2e0f7}4.045 &\cellcolor[HTML]{e0def7}4.307 &\cellcolor[HTML]{dedbf6}4.558 \\
deu\_Latn &\cellcolor[HTML]{e0def7}4.308 &\cellcolor[HTML]{dfddf7}4.379 &\cellcolor[HTML]{dfdcf6}4.445 &\cellcolor[HTML]{dedbf6}4.555 &\cellcolor[HTML]{dcdaf6}4.694 &\cellcolor[HTML]{dad8f5}4.896 &\cellcolor[HTML]{d8d6f5}5.106 & &\cellcolor[HTML]{e7e5f9}3.563 &\cellcolor[HTML]{e5e3f8}3.787 &\cellcolor[HTML]{e4e2f8}3.930 &\cellcolor[HTML]{e2e0f7}4.133 &\cellcolor[HTML]{dfddf7}4.386 &\cellcolor[HTML]{dddaf6}4.669 &\cellcolor[HTML]{dad7f5}4.951 \\
ekk\_Latn &\cellcolor[HTML]{ecebfa}2.993 &\cellcolor[HTML]{e9e7f9}3.384 &\cellcolor[HTML]{e7e5f9}3.562 &\cellcolor[HTML]{e5e3f8}3.786 &\cellcolor[HTML]{e2e0f7}4.072 &\cellcolor[HTML]{dfddf7}4.401 &\cellcolor[HTML]{dcd9f6}4.738 & &\cellcolor[HTML]{f2f1fc}2.435 &\cellcolor[HTML]{ebeafa}3.118 &\cellcolor[HTML]{e9e7f9}3.356 &\cellcolor[HTML]{e6e5f9}3.622 &\cellcolor[HTML]{e3e1f8}3.940 &\cellcolor[HTML]{e0def7}4.296 &\cellcolor[HTML]{dddaf6}4.645 \\
ell\_Grek &\cellcolor[HTML]{dad7f5}4.925 &\cellcolor[HTML]{d5d2f4}5.478 &\cellcolor[HTML]{d1cef3}5.840 &\cellcolor[HTML]{ccc8f1}6.390 &\cellcolor[HTML]{c6c1ef}7.075 &\cellcolor[HTML]{bebaed}7.825 &\cellcolor[HTML]{b7b2ec}8.557 & &\cellcolor[HTML]{f7f7fd}1.853 &\cellcolor[HTML]{dddbf6}4.613 &\cellcolor[HTML]{d7d4f4}5.287 &\cellcolor[HTML]{d0ccf2}6.028 &\cellcolor[HTML]{c7c3f0}6.876 &\cellcolor[HTML]{bfbbee}7.722 &\cellcolor[HTML]{b8b3ec}8.509 \\
eus\_Latn &\cellcolor[HTML]{eae8f9}3.299 &\cellcolor[HTML]{e5e3f8}3.749 &\cellcolor[HTML]{e3e1f8}3.972 &\cellcolor[HTML]{e1def7}4.237 &\cellcolor[HTML]{dedbf6}4.540 &\cellcolor[HTML]{dbd8f5}4.877 &\cellcolor[HTML]{d7d5f4}5.204 & &\cellcolor[HTML]{f1f0fb}2.520 &\cellcolor[HTML]{e9e7f9}3.396 &\cellcolor[HTML]{e6e4f8}3.669 &\cellcolor[HTML]{e3e1f8}3.995 &\cellcolor[HTML]{e0ddf7}4.330 &\cellcolor[HTML]{dcdaf6}4.688 &\cellcolor[HTML]{d9d6f5}5.024 \\
fas\_Arab &\cellcolor[HTML]{d4d1f3}5.585 &\cellcolor[HTML]{d1cdf3}5.915 &\cellcolor[HTML]{cfcbf2}6.136 &\cellcolor[HTML]{ccc8f1}6.445 &\cellcolor[HTML]{c9c5f0}6.767 &\cellcolor[HTML]{c6c2f0}7.050 &\cellcolor[HTML]{c4c0ef}7.257 & &\cellcolor[HTML]{f8f7fd}1.812 &\cellcolor[HTML]{dfddf7}4.406 &\cellcolor[HTML]{d9d7f5}4.997 &\cellcolor[HTML]{d3d0f3}5.689 &\cellcolor[HTML]{cdcaf2}6.281 &\cellcolor[HTML]{c9c5f0}6.712 &\cellcolor[HTML]{c7c3f0}6.948 \\
fin\_Latn &\cellcolor[HTML]{ebe9fa}3.177 &\cellcolor[HTML]{e7e5f9}3.563 &\cellcolor[HTML]{e5e3f8}3.762 &\cellcolor[HTML]{e3e1f8}4.026 &\cellcolor[HTML]{e0ddf7}4.340 &\cellcolor[HTML]{dcdaf6}4.710 &\cellcolor[HTML]{d8d5f5}5.113 & &\cellcolor[HTML]{f2f1fc}2.457 &\cellcolor[HTML]{eae9fa}3.238 &\cellcolor[HTML]{e8e6f9}3.517 &\cellcolor[HTML]{e5e3f8}3.827 &\cellcolor[HTML]{e1dff7}4.191 &\cellcolor[HTML]{dddbf6}4.591 &\cellcolor[HTML]{d9d6f5}5.018 \\
fra\_Latn &\cellcolor[HTML]{dedcf6}4.467 &\cellcolor[HTML]{dedcf6}4.525 &\cellcolor[HTML]{dddbf6}4.567 &\cellcolor[HTML]{dddaf6}4.634 &\cellcolor[HTML]{dcd9f6}4.727 &\cellcolor[HTML]{dbd8f5}4.856 &\cellcolor[HTML]{dad7f5}4.978 & &\cellcolor[HTML]{e7e5f9}3.578 &\cellcolor[HTML]{e5e3f8}3.787 &\cellcolor[HTML]{e4e2f8}3.918 &\cellcolor[HTML]{e2e0f7}4.089 &\cellcolor[HTML]{e0def7}4.290 &\cellcolor[HTML]{dedcf6}4.489 &\cellcolor[HTML]{dcdaf6}4.676 \\
glg\_Latn &\cellcolor[HTML]{e2e0f7}4.069 &\cellcolor[HTML]{e0def7}4.302 &\cellcolor[HTML]{dfddf7}4.405 &\cellcolor[HTML]{dedbf6}4.530 &\cellcolor[HTML]{dcdaf6}4.677 &\cellcolor[HTML]{dbd8f5}4.847 &\cellcolor[HTML]{d9d7f5}5.013 & &\cellcolor[HTML]{eae8fa}3.258 &\cellcolor[HTML]{e6e4f8}3.692 &\cellcolor[HTML]{e4e2f8}3.871 &\cellcolor[HTML]{e2e0f7}4.106 &\cellcolor[HTML]{dfddf7}4.376 &\cellcolor[HTML]{dddaf6}4.645 &\cellcolor[HTML]{dad8f5}4.884 \\
guj\_Gujr &\cellcolor[HTML]{dfddf7}4.369 &\cellcolor[HTML]{ccc8f1}6.388 &\cellcolor[HTML]{c4c0ef}7.225 &\cellcolor[HTML]{bbb6ec}8.217 &\cellcolor[HTML]{b1acea}9.202 &\cellcolor[HTML]{a8a2e7}10.128 &\cellcolor[HTML]{a099e5}11.011 & &1.067 &\cellcolor[HTML]{cecaf2}6.214 &\cellcolor[HTML]{c4c0ef}7.190 &\cellcolor[HTML]{bab5ec}8.291 &\cellcolor[HTML]{b0abea}9.311 &\cellcolor[HTML]{a7a1e7}10.261 &\cellcolor[HTML]{9f98e5}11.101 \\
heb\_Hebr &\cellcolor[HTML]{e3e1f8}4.032 &\cellcolor[HTML]{dedbf6}4.554 &\cellcolor[HTML]{dad8f5}4.886 &\cellcolor[HTML]{d6d3f4}5.315 &\cellcolor[HTML]{d2cef3}5.820 &\cellcolor[HTML]{ccc8f1}6.387 &\cellcolor[HTML]{c6c2f0}6.982 & &\cellcolor[HTML]{f8f7fd}1.790 &\cellcolor[HTML]{e2dff7}4.135 &\cellcolor[HTML]{dddbf6}4.611 &\cellcolor[HTML]{d8d5f5}5.161 &\cellcolor[HTML]{d2cff3}5.744 &\cellcolor[HTML]{ccc9f1}6.354 &\cellcolor[HTML]{c7c3f0}6.958 \\
hin\_Deva &\cellcolor[HTML]{c9c5f0}6.743 &\cellcolor[HTML]{c2beef}7.418 &\cellcolor[HTML]{beb9ed}7.889 &\cellcolor[HTML]{b8b2ec}8.540 &\cellcolor[HTML]{b1acea}9.221 &\cellcolor[HTML]{aba5e8}9.878 &\cellcolor[HTML]{a69fe7}10.436 & &\cellcolor[HTML]{f2f2fc}2.365 &\cellcolor[HTML]{cecaf2}6.202 &\cellcolor[HTML]{c5c1ef}7.142 &\cellcolor[HTML]{bbb7ed}8.135 &\cellcolor[HTML]{b3adea}9.047 &\cellcolor[HTML]{aca6e8}9.783 &\cellcolor[HTML]{a7a0e7}10.330 \\
hrv\_Latn &\cellcolor[HTML]{ebe9fa}3.187 &\cellcolor[HTML]{e8e7f9}3.444 &\cellcolor[HTML]{e7e5f9}3.592 &\cellcolor[HTML]{e5e3f8}3.804 &\cellcolor[HTML]{e2e0f7}4.068 &\cellcolor[HTML]{dfddf7}4.363 &\cellcolor[HTML]{dcdaf6}4.683 & &\cellcolor[HTML]{f1effb}2.571 &\cellcolor[HTML]{eceafa}3.097 &\cellcolor[HTML]{e9e8f9}3.307 &\cellcolor[HTML]{e7e5f9}3.576 &\cellcolor[HTML]{e4e2f8}3.887 &\cellcolor[HTML]{e1dff7}4.225 &\cellcolor[HTML]{dddbf6}4.584 \\
hun\_Latn &\cellcolor[HTML]{e9e7f9}3.404 &\cellcolor[HTML]{e6e5f9}3.643 &\cellcolor[HTML]{e5e3f8}3.817 &\cellcolor[HTML]{e2e0f7}4.072 &\cellcolor[HTML]{dfddf7}4.381 &\cellcolor[HTML]{dcd9f6}4.725 &\cellcolor[HTML]{d8d6f5}5.107 & &\cellcolor[HTML]{efeefb}2.741 &\cellcolor[HTML]{eae9fa}3.239 &\cellcolor[HTML]{e8e6f9}3.503 &\cellcolor[HTML]{e5e3f8}3.822 &\cellcolor[HTML]{e1dff7}4.190 &\cellcolor[HTML]{dddbf6}4.570 &\cellcolor[HTML]{dad7f5}4.970 \\
hye\_Armn &\cellcolor[HTML]{d9d6f5}5.063 &\cellcolor[HTML]{d2cff3}5.755 &\cellcolor[HTML]{cecaf2}6.210 &\cellcolor[HTML]{c8c4f0}6.836 &\cellcolor[HTML]{c1bdee}7.543 &\cellcolor[HTML]{bab5ec}8.288 &\cellcolor[HTML]{b3adea}9.041 & &\cellcolor[HTML]{f9f8fe}1.703 &\cellcolor[HTML]{dbd8f5}4.831 &\cellcolor[HTML]{d4d1f4}5.546 &\cellcolor[HTML]{cdcaf2}6.285 &\cellcolor[HTML]{c6c2f0}7.037 &\cellcolor[HTML]{bfbbee}7.756 &\cellcolor[HTML]{b9b4ec}8.418 \\
ind\_Latn &\cellcolor[HTML]{e4e2f8}3.890 &\cellcolor[HTML]{e0def7}4.266 &\cellcolor[HTML]{dedcf6}4.478 &\cellcolor[HTML]{dcd9f6}4.775 &\cellcolor[HTML]{d9d6f5}5.066 &\cellcolor[HTML]{d6d3f4}5.346 &\cellcolor[HTML]{d4d1f3}5.554 & &\cellcolor[HTML]{f0effb}2.610 &\cellcolor[HTML]{e6e4f8}3.713 &\cellcolor[HTML]{e2e0f7}4.054 &\cellcolor[HTML]{dfdcf6}4.441 &\cellcolor[HTML]{dbd8f5}4.847 &\cellcolor[HTML]{d8d5f4}5.186 &\cellcolor[HTML]{d5d2f4}5.427 \\
isl\_Latn &\cellcolor[HTML]{efeefb}2.737 &\cellcolor[HTML]{e9e8f9}3.313 &\cellcolor[HTML]{e7e6f9}3.532 &\cellcolor[HTML]{e5e3f8}3.793 &\cellcolor[HTML]{e2e0f7}4.095 &\cellcolor[HTML]{dfddf6}4.423 &\cellcolor[HTML]{dcd9f6}4.751 & &\cellcolor[HTML]{f3f2fc}2.341 &\cellcolor[HTML]{ebeafa}3.135 &\cellcolor[HTML]{e9e7f9}3.396 &\cellcolor[HTML]{e6e4f8}3.675 &\cellcolor[HTML]{e3e1f8}3.997 &\cellcolor[HTML]{e0ddf7}4.333 &\cellcolor[HTML]{dcdaf6}4.683 \\
ita\_Latn &\cellcolor[HTML]{e2e0f7}4.053 &\cellcolor[HTML]{e1dff7}4.229 &\cellcolor[HTML]{e0ddf7}4.338 &\cellcolor[HTML]{dedcf6}4.498 &\cellcolor[HTML]{dcdaf6}4.699 &\cellcolor[HTML]{dad8f5}4.891 &\cellcolor[HTML]{d9d6f5}5.077 & &\cellcolor[HTML]{e8e7f9}3.448 &\cellcolor[HTML]{e5e3f8}3.760 &\cellcolor[HTML]{e4e2f8}3.926 &\cellcolor[HTML]{e1dff7}4.149 &\cellcolor[HTML]{dfddf7}4.407 &\cellcolor[HTML]{dcdaf6}4.672 &\cellcolor[HTML]{dad8f5}4.880 \\
jpn\_Jpan &\cellcolor[HTML]{e5e3f8}3.827 &\cellcolor[HTML]{e0def7}4.299 &\cellcolor[HTML]{dedbf6}4.544 &\cellcolor[HTML]{dad8f5}4.881 &\cellcolor[HTML]{d7d4f4}5.292 &\cellcolor[HTML]{d2cff3}5.774 &\cellcolor[HTML]{ccc9f1}6.361 & &\cellcolor[HTML]{f2f1fc}2.408 &\cellcolor[HTML]{eeedfb}2.853 &\cellcolor[HTML]{edecfb}2.895 &\cellcolor[HTML]{edecfa}2.899 &\cellcolor[HTML]{dfddf7}4.394 &\cellcolor[HTML]{d8d5f5}5.143 &\cellcolor[HTML]{d2cff3}5.744 \\
kan\_Knda &\cellcolor[HTML]{cfcbf2}6.111 &\cellcolor[HTML]{c1bcee}7.577 &\cellcolor[HTML]{b9b4ec}8.410 &\cellcolor[HTML]{afa9e9}9.430 &\cellcolor[HTML]{a49de6}10.643 &\cellcolor[HTML]{9790e3}11.914 &\cellcolor[HTML]{8b83df}13.176 & &\cellcolor[HTML]{fefeff}1.115 &\cellcolor[HTML]{c7c3f0}6.886 &\cellcolor[HTML]{bdb8ed}8.013 &\cellcolor[HTML]{b1abea}9.236 &\cellcolor[HTML]{a49de6}10.601 &\cellcolor[HTML]{9790e3}11.953 &\cellcolor[HTML]{8b83df}13.228 \\
kat\_Geor &\cellcolor[HTML]{d1cef3}5.853 &\cellcolor[HTML]{c2beef}7.439 &\cellcolor[HTML]{b9b4ec}8.352 &\cellcolor[HTML]{afa9e9}9.488 &\cellcolor[HTML]{a29be6}10.828 &\cellcolor[HTML]{948de2}12.247 &\cellcolor[HTML]{867ede}13.708 & &\cellcolor[HTML]{f0effb}2.584 &\cellcolor[HTML]{c7c3f0}6.908 &\cellcolor[HTML]{bdb8ed}8.015 &\cellcolor[HTML]{b0aae9}9.365 &\cellcolor[HTML]{a29be6}10.817 &\cellcolor[HTML]{938ce2}12.330 &\cellcolor[HTML]{857dde}13.815 \\
kaz\_Cyrl &\cellcolor[HTML]{dfddf7}4.384 &\cellcolor[HTML]{d1cef3}5.869 &\cellcolor[HTML]{cbc7f1}6.518 &\cellcolor[HTML]{c4c0ef}7.238 &\cellcolor[HTML]{bdb8ed}7.965 &\cellcolor[HTML]{b6b1eb}8.701 &\cellcolor[HTML]{afaae9}9.408 & &\cellcolor[HTML]{ebe9fa}3.199 &\cellcolor[HTML]{d5d2f4}5.504 &\cellcolor[HTML]{cecaf2}6.201 &\cellcolor[HTML]{c6c2f0}7.003 &\cellcolor[HTML]{bfbaee}7.797 &\cellcolor[HTML]{b7b2eb}8.585 &\cellcolor[HTML]{b0aaea}9.329 \\
khk\_Cyrl &\cellcolor[HTML]{e5e3f8}3.824 &\cellcolor[HTML]{d4d1f3}5.567 &\cellcolor[HTML]{cecaf2}6.235 &\cellcolor[HTML]{c6c2f0}7.033 &\cellcolor[HTML]{beb9ed}7.885 &\cellcolor[HTML]{b7b2eb}8.623 &\cellcolor[HTML]{b0abea}9.288 & &\cellcolor[HTML]{edecfa}2.946 &\cellcolor[HTML]{d6d3f4}5.343 &\cellcolor[HTML]{cfccf2}6.076 &\cellcolor[HTML]{c7c3f0}6.908 &\cellcolor[HTML]{bfbaee}7.781 &\cellcolor[HTML]{b7b2ec}8.569 &\cellcolor[HTML]{b1abea}9.272 \\
khm\_Khmr &1.064 &\cellcolor[HTML]{ccc9f1}6.371 &\cellcolor[HTML]{c2beef}7.427 &\cellcolor[HTML]{b8b2ec}8.541 &\cellcolor[HTML]{aea9e9}9.511 &\cellcolor[HTML]{a69fe7}10.412 &\cellcolor[HTML]{9f98e5}11.122 & &\cellcolor[HTML]{fcfcff}1.377 &\cellcolor[HTML]{edecfa}2.906 &\cellcolor[HTML]{cac6f1}6.620 &\cellcolor[HTML]{b9b4ec}8.423 &\cellcolor[HTML]{ada7e9}9.681 &\cellcolor[HTML]{a49de6}10.629 &\cellcolor[HTML]{9d96e4}11.312 \\
kir\_Cyrl &\cellcolor[HTML]{e2e0f7}4.127 &\cellcolor[HTML]{d2cff3}5.787 &\cellcolor[HTML]{ccc8f1}6.404 &\cellcolor[HTML]{c5c1ef}7.101 &\cellcolor[HTML]{bfbaee}7.804 &\cellcolor[HTML]{b7b2ec}8.550 &\cellcolor[HTML]{b1abea}9.271 & &\cellcolor[HTML]{eae8f9}3.285 &\cellcolor[HTML]{d5d2f4}5.468 &\cellcolor[HTML]{cfcbf2}6.120 &\cellcolor[HTML]{c7c3f0}6.878 &\cellcolor[HTML]{c0bbee}7.667 &\cellcolor[HTML]{b9b4ec}8.437 &\cellcolor[HTML]{b1acea}9.201 \\
kor\_Hang &\cellcolor[HTML]{dfdcf6}4.438 &\cellcolor[HTML]{dddbf6}4.581 &\cellcolor[HTML]{dcdaf6}4.695 &\cellcolor[HTML]{dbd8f5}4.875 &\cellcolor[HTML]{d8d5f5}5.132 &\cellcolor[HTML]{d5d2f4}5.489 &\cellcolor[HTML]{d1cef3}5.871 & &\cellcolor[HTML]{faf9fe}1.605 &\cellcolor[HTML]{f2f1fc}2.409 &\cellcolor[HTML]{f2f1fc}2.416 &\cellcolor[HTML]{f2f1fc}2.417 &\cellcolor[HTML]{e4e2f8}3.921 &\cellcolor[HTML]{d9d7f5}5.003 &\cellcolor[HTML]{d3d0f3}5.633 \\
lit\_Latn &\cellcolor[HTML]{edecfb}2.894 &\cellcolor[HTML]{e9e7f9}3.390 &\cellcolor[HTML]{e7e5f9}3.620 &\cellcolor[HTML]{e4e2f8}3.916 &\cellcolor[HTML]{e0def7}4.252 &\cellcolor[HTML]{dddaf6}4.635 &\cellcolor[HTML]{d9d6f5}5.037 & &\cellcolor[HTML]{f2f1fc}2.386 &\cellcolor[HTML]{ebe9fa}3.175 &\cellcolor[HTML]{e8e7f9}3.439 &\cellcolor[HTML]{e5e3f8}3.762 &\cellcolor[HTML]{e2dff7}4.138 &\cellcolor[HTML]{dedbf6}4.551 &\cellcolor[HTML]{dad7f5}4.972 \\
lvs\_Latn &\cellcolor[HTML]{efeefb}2.756 &\cellcolor[HTML]{e9e7f9}3.369 &\cellcolor[HTML]{e7e5f9}3.614 &\cellcolor[HTML]{e4e2f8}3.932 &\cellcolor[HTML]{e0def7}4.284 &\cellcolor[HTML]{dddaf6}4.665 &\cellcolor[HTML]{d9d6f5}5.043 & &\cellcolor[HTML]{f4f3fc}2.258 &\cellcolor[HTML]{ebe9fa}3.156 &\cellcolor[HTML]{e8e7f9}3.442 &\cellcolor[HTML]{e5e3f8}3.773 &\cellcolor[HTML]{e1dff7}4.165 &\cellcolor[HTML]{dddbf6}4.577 &\cellcolor[HTML]{dad7f5}4.979 \\
mal\_Mlym &\cellcolor[HTML]{d2cff3}5.774 &\cellcolor[HTML]{c0bcee}7.625 &\cellcolor[HTML]{b7b2eb}8.602 &\cellcolor[HTML]{aca6e8}9.802 &\cellcolor[HTML]{9f98e5}11.106 &\cellcolor[HTML]{9189e1}12.557 &\cellcolor[HTML]{847bdd}13.990 & &\cellcolor[HTML]{f2f1fc}2.391 &\cellcolor[HTML]{c4c0ef}7.199 &\cellcolor[HTML]{b9b4ec}8.407 &\cellcolor[HTML]{aca6e8}9.727 &\cellcolor[HTML]{9e98e5}11.179 &\cellcolor[HTML]{9088e1}12.705 &\cellcolor[HTML]{8279dd}14.161 \\
mar\_Deva &\cellcolor[HTML]{d0cdf2}5.962 &\cellcolor[HTML]{c0bbee}7.660 &\cellcolor[HTML]{b9b4ec}8.358 &\cellcolor[HTML]{b1abea}9.262 &\cellcolor[HTML]{a7a0e7}10.332 &\cellcolor[HTML]{9d96e4}11.361 &\cellcolor[HTML]{948ce2}12.295 & &\cellcolor[HTML]{f1f0fc}2.496 &\cellcolor[HTML]{cac6f1}6.623 &\cellcolor[HTML]{c0bbee}7.666 &\cellcolor[HTML]{b4afeb}8.895 &\cellcolor[HTML]{a8a2e7}10.171 &\cellcolor[HTML]{9d96e4}11.359 &\cellcolor[HTML]{938ce2}12.342 \\
mkd\_Cyrl &\cellcolor[HTML]{d5d2f4}5.499 &\cellcolor[HTML]{d0ccf2}6.032 &\cellcolor[HTML]{cdc9f1}6.315 &\cellcolor[HTML]{c9c5f0}6.719 &\cellcolor[HTML]{c4c0ef}7.221 &\cellcolor[HTML]{bfbaee}7.806 &\cellcolor[HTML]{b9b4ec}8.383 & &\cellcolor[HTML]{dfddf7}4.401 &\cellcolor[HTML]{d4d1f4}5.533 &\cellcolor[HTML]{d1cdf3}5.926 &\cellcolor[HTML]{ccc8f1}6.428 &\cellcolor[HTML]{c6c2f0}7.040 &\cellcolor[HTML]{bfbbee}7.712 &\cellcolor[HTML]{b9b4ec}8.349 \\
mya\_Mymr &\cellcolor[HTML]{d7d5f4}5.203 &\cellcolor[HTML]{c6c2f0}6.983 &\cellcolor[HTML]{bdb8ed}7.967 &\cellcolor[HTML]{b2adea}9.122 &\cellcolor[HTML]{a69fe7}10.427 &\cellcolor[HTML]{9891e3}11.845 &\cellcolor[HTML]{8a81df}13.345 & &\cellcolor[HTML]{f8f7fd}1.791 &\cellcolor[HTML]{c0bbee}7.665 &\cellcolor[HTML]{b2adea}9.086 &\cellcolor[HTML]{a39de6}10.653 &\cellcolor[HTML]{938be1}12.385 &\cellcolor[HTML]{8279dd}14.195 &\cellcolor[HTML]{7268d8}15.785 \\
nld\_Latn &\cellcolor[HTML]{e5e3f8}3.764 &\cellcolor[HTML]{e3e1f8}4.014 &\cellcolor[HTML]{e1dff7}4.170 &\cellcolor[HTML]{dfddf7}4.360 &\cellcolor[HTML]{dddbf6}4.585 &\cellcolor[HTML]{dbd9f5}4.806 &\cellcolor[HTML]{d9d7f5}4.991 & &\cellcolor[HTML]{eae9fa}3.243 &\cellcolor[HTML]{e6e4f8}3.670 &\cellcolor[HTML]{e4e2f8}3.864 &\cellcolor[HTML]{e2e0f7}4.108 &\cellcolor[HTML]{dfddf7}4.377 &\cellcolor[HTML]{dddaf6}4.639 &\cellcolor[HTML]{dbd8f5}4.865 \\
nno\_Latn &\cellcolor[HTML]{e9e8f9}3.331 &\cellcolor[HTML]{e6e4f8}3.702 &\cellcolor[HTML]{e4e2f8}3.843 &\cellcolor[HTML]{e3e1f8}4.011 &\cellcolor[HTML]{e1dff7}4.218 &\cellcolor[HTML]{dfdcf6}4.434 &\cellcolor[HTML]{dddaf6}4.639 & &\cellcolor[HTML]{eeedfb}2.843 &\cellcolor[HTML]{e8e7f9}3.444 &\cellcolor[HTML]{e6e5f9}3.623 &\cellcolor[HTML]{e5e3f8}3.827 &\cellcolor[HTML]{e2e0f7}4.065 &\cellcolor[HTML]{e0def7}4.314 &\cellcolor[HTML]{dedbf6}4.547 \\
nob\_Latn &\cellcolor[HTML]{e8e7f9}3.431 &\cellcolor[HTML]{e6e4f8}3.689 &\cellcolor[HTML]{e5e3f8}3.828 &\cellcolor[HTML]{e3e1f8}4.005 &\cellcolor[HTML]{e1dff7}4.217 &\cellcolor[HTML]{dfdcf6}4.435 &\cellcolor[HTML]{dddaf6}4.639 & &\cellcolor[HTML]{edecfa}2.902 &\cellcolor[HTML]{e9e7f9}3.408 &\cellcolor[HTML]{e7e5f9}3.585 &\cellcolor[HTML]{e5e3f8}3.803 &\cellcolor[HTML]{e2e0f7}4.049 &\cellcolor[HTML]{e0def7}4.297 &\cellcolor[HTML]{dedbf6}4.542 \\
npi\_Deva &\cellcolor[HTML]{d2cff3}5.730 &\cellcolor[HTML]{c1bdee}7.521 &\cellcolor[HTML]{bbb6ed}8.172 &\cellcolor[HTML]{b2acea}9.130 &\cellcolor[HTML]{a9a2e7}10.121 &\cellcolor[HTML]{9e97e5}11.194 &\cellcolor[HTML]{958ee2}12.158 & &\cellcolor[HTML]{f2f1fc}2.433 &\cellcolor[HTML]{cbc7f1}6.545 &\cellcolor[HTML]{c0bbee}7.659 &\cellcolor[HTML]{b4afeb}8.890 &\cellcolor[HTML]{a8a2e7}10.141 &\cellcolor[HTML]{9e97e4}11.273 &\cellcolor[HTML]{948ce2}12.271 \\
ory\_Orya &1.010 &\cellcolor[HTML]{cbc7f1}6.522 &\cellcolor[HTML]{c0bbee}7.674 &\cellcolor[HTML]{b4afeb}8.922 &\cellcolor[HTML]{a8a2e7}10.176 &\cellcolor[HTML]{9d96e4}11.318 &\cellcolor[HTML]{948ce2}12.272 & &1.006 &\cellcolor[HTML]{c9c5f0}6.715 &\cellcolor[HTML]{beb9ed}7.874 &\cellcolor[HTML]{b2adea}9.125 &\cellcolor[HTML]{a6a0e7}10.376 &\cellcolor[HTML]{9c95e4}11.422 &\cellcolor[HTML]{948ce2}12.290 \\
pan\_Guru &\cellcolor[HTML]{e1dff7}4.168 &\cellcolor[HTML]{cdc9f1}6.319 &\cellcolor[HTML]{c6c2f0}7.047 &\cellcolor[HTML]{bebaed}7.832 &\cellcolor[HTML]{b8b3ec}8.539 &\cellcolor[HTML]{b2acea}9.131 &\cellcolor[HTML]{aea8e9}9.562 & &\cellcolor[HTML]{fefeff}1.172 &\cellcolor[HTML]{cfccf2}6.070 &\cellcolor[HTML]{c8c4f0}6.831 &\cellcolor[HTML]{c0bcee}7.623 &\cellcolor[HTML]{bab5ec}8.312 &\cellcolor[HTML]{b5b0eb}8.831 &\cellcolor[HTML]{b1acea}9.198 \\
pol\_Latn &\cellcolor[HTML]{e9e7f9}3.382 &\cellcolor[HTML]{e6e4f9}3.644 &\cellcolor[HTML]{e5e3f8}3.807 &\cellcolor[HTML]{e3e1f8}4.024 &\cellcolor[HTML]{e0def7}4.311 &\cellcolor[HTML]{dddaf6}4.651 &\cellcolor[HTML]{d9d6f5}5.038 & &\cellcolor[HTML]{eeedfb}2.809 &\cellcolor[HTML]{eae8fa}3.256 &\cellcolor[HTML]{e8e6f9}3.497 &\cellcolor[HTML]{e5e3f8}3.789 &\cellcolor[HTML]{e2dff7}4.141 &\cellcolor[HTML]{dedcf6}4.519 &\cellcolor[HTML]{dad7f5}4.933 \\
por\_Latn &\cellcolor[HTML]{e1dff7}4.169 &\cellcolor[HTML]{e0def7}4.292 &\cellcolor[HTML]{dfddf7}4.375 &\cellcolor[HTML]{dedcf6}4.491 &\cellcolor[HTML]{dddaf6}4.643 &\cellcolor[HTML]{dbd9f5}4.816 &\cellcolor[HTML]{dad7f5}4.973 & &\cellcolor[HTML]{e9e8f9}3.349 &\cellcolor[HTML]{e6e4f8}3.697 &\cellcolor[HTML]{e4e2f8}3.869 &\cellcolor[HTML]{e2e0f7}4.091 &\cellcolor[HTML]{e0ddf7}4.340 &\cellcolor[HTML]{dddbf6}4.608 &\cellcolor[HTML]{dbd8f5}4.851 \\
ron\_Latn &\cellcolor[HTML]{e8e6f9}3.510 &\cellcolor[HTML]{e4e2f8}3.854 &\cellcolor[HTML]{e3e1f8}4.015 &\cellcolor[HTML]{e1dff7}4.217 &\cellcolor[HTML]{dfdcf6}4.460 &\cellcolor[HTML]{dcdaf6}4.721 &\cellcolor[HTML]{dad7f5}4.956 & &\cellcolor[HTML]{edebfa}2.972 &\cellcolor[HTML]{e7e5f9}3.554 &\cellcolor[HTML]{e5e3f8}3.746 &\cellcolor[HTML]{e3e1f8}3.981 &\cellcolor[HTML]{e0def7}4.252 &\cellcolor[HTML]{dedbf6}4.534 &\cellcolor[HTML]{dbd9f5}4.801 \\
rus\_Cyrl &\cellcolor[HTML]{cecaf2}6.220 &\cellcolor[HTML]{ccc8f1}6.443 &\cellcolor[HTML]{cac6f1}6.606 &\cellcolor[HTML]{c8c4f0}6.859 &\cellcolor[HTML]{c4c0ef}7.224 &\cellcolor[HTML]{bfbbee}7.729 &\cellcolor[HTML]{bab5ec}8.317 & &\cellcolor[HTML]{d8d5f5}5.167 &\cellcolor[HTML]{d3d0f3}5.671 &\cellcolor[HTML]{d0cdf2}5.966 &\cellcolor[HTML]{ccc9f1}6.382 &\cellcolor[HTML]{c7c3f0}6.886 &\cellcolor[HTML]{c1bdee}7.515 &\cellcolor[HTML]{bbb6ec}8.204 \\
sin\_Sinh &1.010 &\cellcolor[HTML]{cfccf2}6.048 &\cellcolor[HTML]{c7c3f0}6.922 &\cellcolor[HTML]{bdb9ed}7.945 &\cellcolor[HTML]{b4afeb}8.930 &\cellcolor[HTML]{aca6e8}9.770 &\cellcolor[HTML]{a59fe6}10.491 & &\cellcolor[HTML]{fbfbfe}1.431 &\cellcolor[HTML]{cfccf2}6.084 &\cellcolor[HTML]{c7c3f0}6.973 &\cellcolor[HTML]{bdb8ed}8.016 &\cellcolor[HTML]{b3aeea}9.016 &\cellcolor[HTML]{aba5e8}9.864 &\cellcolor[HTML]{a49ee6}10.565 \\
slk\_Latn &\cellcolor[HTML]{eceafa}3.093 &\cellcolor[HTML]{e8e6f9}3.506 &\cellcolor[HTML]{e6e4f8}3.683 &\cellcolor[HTML]{e4e2f8}3.917 &\cellcolor[HTML]{e1dff7}4.209 &\cellcolor[HTML]{dedbf6}4.552 &\cellcolor[HTML]{dad7f5}4.917 & &\cellcolor[HTML]{f1f0fb}2.548 &\cellcolor[HTML]{ebe9fa}3.166 &\cellcolor[HTML]{e8e7f9}3.413 &\cellcolor[HTML]{e6e4f8}3.719 &\cellcolor[HTML]{e2e0f7}4.045 &\cellcolor[HTML]{dfdcf6}4.429 &\cellcolor[HTML]{dbd9f5}4.811 \\
slv\_Latn &\cellcolor[HTML]{eceafa}3.088 &\cellcolor[HTML]{e9e7f9}3.385 &\cellcolor[HTML]{e7e6f9}3.539 &\cellcolor[HTML]{e5e4f8}3.738 &\cellcolor[HTML]{e3e1f8}3.980 &\cellcolor[HTML]{e0def7}4.255 &\cellcolor[HTML]{dedbf6}4.555 & &\cellcolor[HTML]{f0effb}2.595 &\cellcolor[HTML]{ebeafa}3.107 &\cellcolor[HTML]{e9e8f9}3.314 &\cellcolor[HTML]{e7e5f9}3.566 &\cellcolor[HTML]{e4e2f8}3.844 &\cellcolor[HTML]{e2dff7}4.143 &\cellcolor[HTML]{dedcf6}4.472 \\
spa\_Latn &\cellcolor[HTML]{dfddf7}4.367 &\cellcolor[HTML]{dfdcf6}4.450 &\cellcolor[HTML]{dedcf6}4.505 &\cellcolor[HTML]{dddbf6}4.592 &\cellcolor[HTML]{dcdaf6}4.710 &\cellcolor[HTML]{dbd8f5}4.848 &\cellcolor[HTML]{dad7f5}4.979 & &\cellcolor[HTML]{e7e5f9}3.548 &\cellcolor[HTML]{e5e3f8}3.812 &\cellcolor[HTML]{e3e1f8}3.966 &\cellcolor[HTML]{e1dff7}4.170 &\cellcolor[HTML]{dfddf7}4.411 &\cellcolor[HTML]{dddaf6}4.662 &\cellcolor[HTML]{dbd8f5}4.871 \\
srp\_Cyrl &\cellcolor[HTML]{d7d4f4}5.215 &\cellcolor[HTML]{d3d0f3}5.689 &\cellcolor[HTML]{d0cdf2}5.949 &\cellcolor[HTML]{cdc9f2}6.307 &\cellcolor[HTML]{c9c5f0}6.739 &\cellcolor[HTML]{c3bfef}7.295 &\cellcolor[HTML]{beb9ed}7.878 & &\cellcolor[HTML]{dfdcf6}4.440 &\cellcolor[HTML]{d7d4f4}5.233 &\cellcolor[HTML]{d4d1f3}5.600 &\cellcolor[HTML]{cfccf2}6.037 &\cellcolor[HTML]{cac6f1}6.589 &\cellcolor[HTML]{c4c0ef}7.213 &\cellcolor[HTML]{beb9ed}7.860 \\
swe\_Latn &\cellcolor[HTML]{e8e6f9}3.511 &\cellcolor[HTML]{e5e4f8}3.727 &\cellcolor[HTML]{e4e2f8}3.860 &\cellcolor[HTML]{e2e0f7}4.059 &\cellcolor[HTML]{e0def7}4.308 &\cellcolor[HTML]{dddbf6}4.592 &\cellcolor[HTML]{dbd8f5}4.830 & &\cellcolor[HTML]{eceafa}3.068 &\cellcolor[HTML]{e8e7f9}3.439 &\cellcolor[HTML]{e7e5f9}3.616 &\cellcolor[HTML]{e4e2f8}3.850 &\cellcolor[HTML]{e2e0f7}4.127 &\cellcolor[HTML]{dfdcf6}4.432 &\cellcolor[HTML]{dcdaf6}4.713 \\
swh\_Latn &\cellcolor[HTML]{efeefb}2.736 &\cellcolor[HTML]{e8e6f9}3.514 &\cellcolor[HTML]{e5e3f8}3.794 &\cellcolor[HTML]{e2e0f7}4.114 &\cellcolor[HTML]{dfdcf6}4.442 &\cellcolor[HTML]{dcdaf6}4.717 &\cellcolor[HTML]{dad7f5}4.928 & &\cellcolor[HTML]{f2f1fc}2.398 &\cellcolor[HTML]{e9e8f9}3.315 &\cellcolor[HTML]{e6e5f9}3.626 &\cellcolor[HTML]{e3e1f8}3.976 &\cellcolor[HTML]{e0ddf7}4.337 &\cellcolor[HTML]{dddaf6}4.631 &\cellcolor[HTML]{dbd8f5}4.860 \\
tam\_Taml &\cellcolor[HTML]{c7c3f0}6.956 &\cellcolor[HTML]{b5b0eb}8.778 &\cellcolor[HTML]{aca6e8}9.735 &\cellcolor[HTML]{a099e5}11.009 &\cellcolor[HTML]{948ce2}12.314 &\cellcolor[HTML]{867ede}13.703 &\cellcolor[HTML]{7a71db}14.998 & &\cellcolor[HTML]{f3f2fc}2.311 &\cellcolor[HTML]{c0bbee}7.708 &\cellcolor[HTML]{b2acea}9.136 &\cellcolor[HTML]{a39ce6}10.730 &\cellcolor[HTML]{948de2}12.253 &\cellcolor[HTML]{867dde}13.776 &\cellcolor[HTML]{7970da}15.089 \\
tel\_Telu &\cellcolor[HTML]{d3d0f3}5.623 &\cellcolor[HTML]{c3bfef}7.353 &\cellcolor[HTML]{bbb6ed}8.180 &\cellcolor[HTML]{b2acea}9.160 &\cellcolor[HTML]{a7a1e7}10.292 &\cellcolor[HTML]{9b94e4}11.538 &\cellcolor[HTML]{9088e1}12.654 & &1.066 &\cellcolor[HTML]{c9c5f0}6.712 &\cellcolor[HTML]{bfbaee}7.757 &\cellcolor[HTML]{b3aeea}8.985 &\cellcolor[HTML]{a6a0e7}10.343 &\cellcolor[HTML]{9a93e3}11.651 &\cellcolor[HTML]{8f87e0}12.769 \\
tha\_Thai &\cellcolor[HTML]{d4d1f4}5.545 &\cellcolor[HTML]{bfbaee}7.767 &\cellcolor[HTML]{b4aeeb}8.958 &\cellcolor[HTML]{a7a1e7}10.228 &\cellcolor[HTML]{9a92e3}11.686 &\cellcolor[HTML]{8c84e0}13.105 &\cellcolor[HTML]{7e75dc}14.551 & &\cellcolor[HTML]{efeefb}2.696 &\cellcolor[HTML]{cdc9f1}6.347 &\cellcolor[HTML]{bdb8ed}8.000 &\cellcolor[HTML]{aea8e9}9.581 &\cellcolor[HTML]{a099e5}11.036 &\cellcolor[HTML]{938be1}12.387 &\cellcolor[HTML]{877ede}13.672 \\
tur\_Latn &\cellcolor[HTML]{e7e5f9}3.564 &\cellcolor[HTML]{e4e2f8}3.899 &\cellcolor[HTML]{e2e0f7}4.119 &\cellcolor[HTML]{dfddf7}4.408 &\cellcolor[HTML]{dcd9f6}4.741 &\cellcolor[HTML]{d8d6f5}5.111 &\cellcolor[HTML]{d5d2f4}5.487 & &\cellcolor[HTML]{f3f3fc}2.271 &\cellcolor[HTML]{e9e7f9}3.364 &\cellcolor[HTML]{e6e4f8}3.685 &\cellcolor[HTML]{e2e0f7}4.083 &\cellcolor[HTML]{dedcf6}4.508 &\cellcolor[HTML]{dad8f5}4.907 &\cellcolor[HTML]{d7d4f4}5.296 \\
ukr\_Cyrl &\cellcolor[HTML]{d8d5f5}5.115 &\cellcolor[HTML]{d3d0f3}5.629 &\cellcolor[HTML]{d1cdf3}5.927 &\cellcolor[HTML]{cdc9f1}6.315 &\cellcolor[HTML]{c8c4f0}6.816 &\cellcolor[HTML]{c2beef}7.414 &\cellcolor[HTML]{bcb7ed}8.085 & &\cellcolor[HTML]{dddbf6}4.614 &\cellcolor[HTML]{d7d4f4}5.267 &\cellcolor[HTML]{d3d0f3}5.618 &\cellcolor[HTML]{cfccf2}6.083 &\cellcolor[HTML]{c9c6f1}6.667 &\cellcolor[HTML]{c3bfef}7.317 &\cellcolor[HTML]{bdb8ed}8.010 \\
urd\_Arab &\cellcolor[HTML]{dcd9f6}4.746 &\cellcolor[HTML]{d5d2f4}5.445 &\cellcolor[HTML]{d2cff3}5.754 &\cellcolor[HTML]{cfcbf2}6.098 &\cellcolor[HTML]{cbc8f1}6.454 &\cellcolor[HTML]{c8c5f0}6.769 &\cellcolor[HTML]{c6c2f0}7.004 & &\cellcolor[HTML]{f9f8fe}1.712 &\cellcolor[HTML]{dedcf6}4.514 &\cellcolor[HTML]{d8d5f5}5.140 &\cellcolor[HTML]{d1cef3}5.854 &\cellcolor[HTML]{cbc8f1}6.463 &\cellcolor[HTML]{c7c3f0}6.925 &\cellcolor[HTML]{c4c0ef}7.225 \\
uzn\_Latn &\cellcolor[HTML]{edecfa}2.953 &\cellcolor[HTML]{e7e6f9}3.529 &\cellcolor[HTML]{e5e3f8}3.786 &\cellcolor[HTML]{e2e0f7}4.090 &\cellcolor[HTML]{dfdcf6}4.440 &\cellcolor[HTML]{dbd9f5}4.797 &\cellcolor[HTML]{d8d6f5}5.112 & &\cellcolor[HTML]{f3f2fc}2.363 &\cellcolor[HTML]{ebe9fa}3.181 &\cellcolor[HTML]{e8e6f9}3.470 &\cellcolor[HTML]{e5e3f8}3.796 &\cellcolor[HTML]{e2e0f7}4.132 &\cellcolor[HTML]{dfdcf6}4.447 &\cellcolor[HTML]{dcd9f6}4.728 \\
vie\_Latn &\cellcolor[HTML]{dddbf6}4.613 &\cellcolor[HTML]{dad7f5}4.947 &\cellcolor[HTML]{d9d6f5}5.083 &\cellcolor[HTML]{d8d5f4}5.174 &\cellcolor[HTML]{d7d4f4}5.219 &\cellcolor[HTML]{d7d4f4}5.239 &\cellcolor[HTML]{d7d4f4}5.249 & &\cellcolor[HTML]{f6f5fd}2.026 &\cellcolor[HTML]{e1dff7}4.168 &\cellcolor[HTML]{dcd9f6}4.767 &\cellcolor[HTML]{d9d6f5}5.061 &\cellcolor[HTML]{d8d5f5}5.146 &\cellcolor[HTML]{d8d5f4}5.184 &\cellcolor[HTML]{d7d4f4}5.211 \\
zsm\_Latn &\cellcolor[HTML]{e4e2f8}3.910 &\cellcolor[HTML]{dfddf7}4.409 &\cellcolor[HTML]{dddaf6}4.668 &\cellcolor[HTML]{dad7f5}4.969 &\cellcolor[HTML]{d7d4f4}5.296 &\cellcolor[HTML]{d4d1f3}5.557 &\cellcolor[HTML]{d2cff3}5.731 & &\cellcolor[HTML]{f0effb}2.594 &\cellcolor[HTML]{e5e3f8}3.803 &\cellcolor[HTML]{e1dff7}4.188 &\cellcolor[HTML]{dddbf6}4.621 &\cellcolor[HTML]{d9d6f5}5.038 &\cellcolor[HTML]{d6d3f4}5.369 &\cellcolor[HTML]{d4d1f3}5.573 \\
\bottomrule
\end{tabular}

References

Section Summary: This section compiles a bibliography of research papers, technical reports, and tools focused on tokenization methods in machine translation and large language models. The works examine techniques like subword segmentation to manage rare words, adapt models to new or underrepresented languages, and address issues such as vocabulary efficiency and performance gaps across different linguistic structures. Many entries analyze real-world impacts on model fairness, compression, and adaptation, drawing from conferences and arXiv preprints spanning several decades.

[1] Sennrich et al. (2016). Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1715–1725. doi:10.18653/v1/P16-1162. https://aclanthology.org/P16-1162/.

[2] Gage, Philip (1994). A new algorithm for data compression. The C Users Journal. 12(2). pp. 23–38.

[3] Petrov et al. (2023). Language Model Tokenizers Introduce Unfairness Between Languages. In Advances in Neural Information Processing Systems. pp. 36963–36990. https://proceedings.neurips.cc/paper_files/paper/2023/file/74bb24dca8334adce292883b4b651eda-Paper-Conference.pdf.

[4] Tejaswi et al. (2024). Exploring Design Choices for Building Language-Specific LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024. pp. 10485–10500. doi:10.18653/v1/2024.findings-emnlp.614. https://aclanthology.org/2024.findings-emnlp.614/.

[5] Csaki et al. (2024). SambaLingo: Teaching Large Language Models New Languages. In Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024). pp. 1–21. doi:10.18653/v1/2024.mrl-1.1. https://aclanthology.org/2024.mrl-1.1/.

[6] Zoltan Csaki et al. (2023). Efficiently Adapting Pretrained Language Models To New Languages. https://arxiv.org/abs/2311.05741. arXiv:2311.05741.

[7] Kiulian et al. (2025). From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages. In Proceedings of the Fourth Ukrainian Natural Language Processing Workshop (UNLP 2025). pp. 1–13. doi:10.18653/v1/2025.unlp-1.1. https://aclanthology.org/2025.unlp-1.1/.

[8] Kazuki Fujii et al. (2024). Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities. In First Conference on Language Modeling. https://openreview.net/forum?id=TQdd1VhWbe.

[9] Peiqin Lin et al. (2024). MaLA-500: Massive Language Adaptation of Large Language Models. https://arxiv.org/abs/2401.13303. arXiv:2401.13303.

[10] Atsuki Yamaguchi et al. (2024). How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?. https://arxiv.org/abs/2406.11477. arXiv:2406.11477.

[11] Yiming Cui et al. (2024). Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca. https://arxiv.org/abs/2304.08177. arXiv:2304.08177.

[12] Aishwarya Kamath et al. (2025). Gemma 3 Technical Report. https://arxiv.org/abs/2503.19786. arXiv:2503.19786.

[13] Olivier Lacombe et al. (2025). Introducing Gemma 3 270M: The compact model for hyper-efficient AI. Google Developers Blog. Accessed: 2026-01-16. https://developers.googleblog.com/en/introducing-gemma-3-270m/.

[14] Thomas Wolf et al. (2020). Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pp. 38–45. https://www.aclweb.org/anthology/2020.emnlp-demos.6.

[15] Rust et al. (2021). How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 3118–3135. doi:10.18653/v1/2021.acl-long.243. https://aclanthology.org/2021.acl-long.243/.

[16] Goldman et al. (2024). Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance. In Findings of the Association for Computational Linguistics: ACL 2024. pp. 2274–2286. doi:10.18653/v1/2024.findings-acl.134. https://aclanthology.org/2024.findings-acl.134/.

[17] Arnett, Catherine and Bergen, Benjamin (2025). Why do language models perform worse for morphologically complex languages?. In Proceedings of the 31st International Conference on Computational Linguistics. pp. 6607–6623. https://aclanthology.org/2025.coling-main.441/.

[18] Balde et al. (2025). Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settings. In Findings of the Association for Computational Linguistics: ACL 2025. pp. 22989–23004. doi:10.18653/v1/2025.findings-acl.1179. https://aclanthology.org/2025.findings-acl.1179/.

[19] Balde et al. (2024). Adaptive BPE Tokenization for Enhanced Vocabulary Adaptation in Finetuning Pretrained Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2024. pp. 14724–14733. doi:10.18653/v1/2024.findings-emnlp.863. https://aclanthology.org/2024.findings-emnlp.863/.

[20] Yang et al. (2022). TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. pp. 35–43. doi:10.18653/v1/2022.acl-demo.4. https://aclanthology.org/2022.acl-demo.4/.

[21] Cognetta et al. (2024). An Analysis of BPE Vocabulary Trimming in Neural Machine Translation. In Proceedings of the Fifth Workshop on Insights from Negative Results in NLP. pp. 48–50. doi:10.18653/v1/2024.insights-1.7. https://aclanthology.org/2024.insights-1.7/.

[22] Bauwens, Thomas and Delobelle, Pieter (2024). BPE-knockout: Pruning Pre-existing BPE Tokenisers with Backwards-compatible Morphological Semi-supervision. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). pp. 5810–5832. doi:10.18653/v1/2024.naacl-long.324. https://aclanthology.org/2024.naacl-long.324/.

[23] Chizhov et al. (2024). BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 16587–16604. doi:10.18653/v1/2024.emnlp-main.925. https://aclanthology.org/2024.emnlp-main.925/.

[24] Lian et al. (2025). Scaffold-bpe: Enhancing byte pair encoding for large language models with simple and effective scaffold token removal. In Proceedings of the AAAI Conference on Artificial Intelligence. pp. 24539–24548.

[25] Samuel et al. (2025). Small Languages, Big Models: A Study of Continual Training on Languages of Norway. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025). pp. 573–608. https://aclanthology.org/2025.nodalida-1.61/.

[26] Gautier Dagan et al. (2024). Getting the most out of your tokenizer for pre-training and domain adaptation. In Forty-first International Conference on Machine Learning. https://openreview.net/forum?id=ZFYBnLljtT.

[27] Gee et al. (2022). Fast Vocabulary Transfer for Language Model Compression. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track. pp. 409–416. https://aclanthology.org/2022.emnlp-industry.41.

[28] Dobler, Konstantin and de Melo, Gerard (2023). FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 13440–13454. doi:10.18653/v1/2023.emnlp-main.829. https://aclanthology.org/2023.emnlp-main.829/.

[29] Radford et al. (2019). Language models are unsupervised multitask learners. OpenAI blog. 1(8). pp. 9.

[30] Kudo, Taku and Richardson, John (2018). SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pp. 66–71. doi:10.18653/v1/D18-2012. https://aclanthology.org/D18-2012/.

[31] Aaron Grattafiori et al. (2024). The Llama 3 Herd of Models. https://arxiv.org/abs/2407.21783. arXiv:2407.21783.

[32] Land, Sander and Bartolo, Max (2024). Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 11631–11646.

[33] Guilherme Penedo et al. (2025). FineWeb2: One Pipeline to Scale Them All — Adapting Pre-Training Data Processing to Every Language. In Second Conference on Language Modeling. https://openreview.net/forum?id=jnRBe6zatP.

[34] Penedo et al. (2024). The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In Advances in Neural Information Processing Systems. pp. 30811–30849. doi:10.52202/079017-0970. https://proceedings.neurips.cc/paper_files/paper/2024/file/370df50ccfdf8bde18f8f9c2d9151bda-Paper-Datasets_and_Benchmarks_Track.pdf.

[35] Costa-Jussà et al. (2022). No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672.

[36] Zouhar et al. (2023). Tokenization and the Noiseless Channel. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 5184–5207. doi:10.18653/v1/2023.acl-long.284. https://aclanthology.org/2023.acl-long.284/.

[37] Team NLLB et al. (2022). No Language Left Behind: Scaling Human-Centered Machine Translation. ArXiv. abs/2207.04672. https://api.semanticscholar.org/CorpusID:250425961.

[38] Sakaguchi et al. (2021). Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM. 64(9). pp. 99–106.

[39] Marii Ojastu et al. (2025). Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translation. https://arxiv.org/abs/2511.17290. arXiv:2511.17290.

[40] Ponti et al. (2020). XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 2362–2376. doi:10.18653/v1/2020.emnlp-main.185. https://aclanthology.org/2020.emnlp-main.185/.

[41] Adelani et al. (2024). SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 226–245. doi:10.18653/v1/2024.eacl-long.14. https://aclanthology.org/2024.eacl-long.14/.

[42] Bandarkar et al. (2024). The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 749–775. https://aclanthology.org/2024.acl-long.44.

[43] Gao et al. (2023). A framework for few-shot language model evaluation. doi:10.5281/zenodo.10256836. https://zenodo.org/records/10256836.

[44] Rei et al. (2020). COMET: A Neural Framework for MT Evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 2685–2702. doi:10.18653/v1/2020.emnlp-main.213. https://aclanthology.org/2020.emnlp-main.213/.

[45] Rei et al. (2022). COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task. In Proceedings of the Seventh Conference on Machine Translation (WMT). pp. 578–585. https://aclanthology.org/2022.wmt-1.52/.

[46] Hugo Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. https://arxiv.org/abs/2307.09288. arXiv:2307.09288.

[47] Qwen et al. (2025). Qwen2.5 Technical Report. https://arxiv.org/abs/2412.15115. arXiv:2412.15115.