When is Routing Meaningful? Diversity and Robustness in Language Model Societies

Fantine Huot Michael Kaisers Mirella Lapata
Google DeepMind
{fantinehuot, mkaisers, lapata}@google.com

Abstract

Routing policies for multi-model systems are evaluated almost exclusively on task accuracy and inference cost. We argue that two properties, orthogonal to performance, determine whether routing is meaningful. First, the society of actors must be behaviourally differentiated: if all actors respond identically, routing is vacuous. Second, the routing policy must be stable: surface-form variants of a query should be assigned to the same actor. High task accuracy is compatible with violating both properties, since a router can operate over a redundant society or assign queries inconsistently, preventing specialisation regardless of performance. We adapt Hierarchic Social Entropy (HSE) to language-model societies and introduce a perturbation-based robustness metric to diagnose these failure modes. Applied to EmbedLLM and RouterBench, we find that HSE exhibits strong diminishing returns, suggesting that a curated subset of fewer than ten agents recovers most available diversity in a large pool — a practical coreset heuristic for society design. We further find that KNN routers gain accuracy from specialist societies but collapse in robustness under perturbation, while prompted routing remains stable across all perturbation types — illustrating that accuracy and meaningfulness can sharply diverge.

Executive Summary: Routing policies for multi-model language systems are typically judged only on task accuracy and inference cost. In practice, however, routing is meaningful only when the collection of models (a “society”) contains genuinely different behavioural profiles and the routing policy assigns semantically equivalent queries to the same model every time. Without these two properties, even a high-accuracy router cannot produce reliable specialisation or stable task distributions.

The paper therefore evaluates two structural requirements for meaningful routing. First, it measures behavioural diversity across real-world and synthetic model societies. Second, it measures whether routing decisions remain stable when queries are varied by spelling, wording, syntax, paraphrasing or added context.

The authors adapt an existing information-theoretic diversity metric to language-model outputs, compute it on two public routing benchmarks (one with 112 models, one with 11), and introduce a five-level perturbation test that tracks whether each router keeps variants of the same query assigned to one model. They compare a default pool of publicly available models against purpose-built specialist societies and against small greedily selected subsets.

The analysis yields four main results. Specialist societies are roughly twice as diverse as existing model pools of the same size. Diversity saturates rapidly: nine or fewer models recover most of the available behavioural spread in the larger benchmark, and four suffice in the smaller one. KNN-style routers achieve their highest accuracy on the most diverse societies yet suffer the largest drops in robustness under perturbation. Prompt-based routers show the opposite pattern—modestly lower accuracy but consistently stable assignment across all perturbation types and society configurations.

These patterns indicate that accuracy and robustness can move in opposite directions. A router that looks optimal on clean queries may scatter work across models once inputs vary, preventing the stable task distributions required for specialisation. Conversely, a modest increase in diversity combined with a stable policy can preserve performance while improving consistency.

Organisations building routed systems should therefore (a) select small, curated model subsets that maximise measured diversity rather than simply adding more models, (b) favour routing methods whose decisions remain stable under surface variation, and (c) report both diversity and robustness alongside accuracy when evaluating new policies. When further validation is needed, joint optimisation of diversity and accuracy on a held-out set provides a practical next step.

The findings rest on two simplifications: synthetic specialists have perfectly binary, non-overlapping expertise, and robustness is measured only on single-turn assignment rather than multi-step agent behaviour. Results on real, softer expert profiles may therefore be somewhat less extreme, but the central trade-off between diversity-driven accuracy and perturbation stability appears robust across the two benchmarks examined.

1. Introduction

Section Summary: Existing evaluations of LLM routing focus on task performance like accuracy and cost, yet overlook whether a system of models actually contains meaningfully different behaviors and whether routing decisions remain stable under minor wording changes. To address this gap, the authors adapt Hierarchic Social Entropy to quantify behavioral diversity among models and propose a perturbation-based metric to test routing robustness, validating both on established benchmarks against both real-world and synthetic model pools. Their results show that real-world models exhibit surprisingly low diversity with strong diminishing returns beyond a small number of agents, and that high diversity does not guarantee robustness.

Routing is becoming a central mechanism for coordinating systems composed of multiple language models, tools, or agents ([1, 2, 3, 4]). Existing work on LLM routing largely evaluates routers by downstream utility: whether they improve task accuracy, reduce cost, or select the best-performing model for a query ([5, 6, 7, 8]). A router is not merely a predictor of task performance; it is also the coordination mechanism that determines how work is distributed across a society of actors. Its usefulness, therefore, depends on two structural properties. First, the society must contain behaviourally differentiated actors: if all actors respond in the same way, routing is vacuous ([9, 10]). Second, the routing policy should be robust to superficial variation: semantically-equivalent variants should not be sent to different actors merely because of changes in spelling, syntax, or wording ([11]). Without such stability, actors cannot reliably specialise.

To characterise the space of choices available to a router, we adapt Hierarchic Social Entropy ([9]) to language-model societies, measuring behavioural diversity from model outputs rather than parameters ([12]). To characterise the stability of the router's choices, we introduce a perturbation-based robustness metric that measures whether semantically related inputs are assigned to the same actor.

We validate HSE on two benchmarks, EmbedLLM ([13]) and RouterBench ([7]), comparing a default society of real-world language models against synthetic societies of purpose-designed experts varying in role and expertise overlap.

We find that specialist societies achieve substantially higher HSE than pools of real-world models of equivalent size, suggesting that behavioural diversity in deployed model pools is lower than commonly assumed. We further observe strong diminishing returns: fewer than ten agents suffice to capture most available diversity in EmbedLLM, and four agents in RouterBench, providing a practical heuristic for society design. Interestingly, higher HSE does not imply higher robustness: KNN routers achieve their best accuracy on specialist societies but their worst robustness, while prompted routing remains stable across all HSE levels and perturbation types. Our contributions are summarised as follows:

• We introduce the first behavioural diversity metric for LM routing analysis, adapting Hierarchic Social Entropy to language-model societies to characterise when routing is meaningful.
• We introduce a perturbation-based metric that quantifies how stably a routing policy assigns semantically-equivalent queries to the same actor, paired with a five-level perturbation taxonomy for fine-grained analysis by perturbation type.
• We show that HSE exhibits strong diminishing returns with society size: a curated subset of fewer than ten agents recovers most available diversity in a large real-world pool, providing a practical coreset heuristic for society design.

2. Related Work

Section Summary: Recent research on LLM routing examines ways to direct queries among multiple models to balance accuracy against cost and speed, with extensions into multi-agent setups that also assign roles and collaboration patterns. Parallel work from robotics has developed metrics to quantify behavioral diversity and role specialization within groups of agents, ideas that are now being adapted to language-model systems. A separate thread studies how models respond to small input changes such as paraphrases, but the focus here shifts from output stability to whether routing decisions themselves remain consistent under such perturbations.

LLM Routing and Model Selection

A growing body of work studies routing as a mechanism for selecting among language models with different capabilities, costs, and latency profiles. In this setting, the router is typically evaluated by its ability to improve the trade-off between task performance and inference cost. FrugalGPT ([5]), for example, formulates LLM use as a cost-sensitive cascade, where cheaper models are queried before more expensive ones in order to reduce inference cost while maintaining task accuracy. RouterBench ([7]) provides a benchmark for multi-LLM routing, evaluating how well routing policies choose among alternative models across tasks and model families. RouteLLM ([6]) learns routing policies from preference data, routing queries between stronger and weaker models in order to preserve response quality while reducing the use of expensive models.

More recently, MasRouter ([4]) extends routing to multi-agent settings, jointly optimising collaboration mode, role allocation, and model selection, building on multi-agent LLM frameworks in which agent roles and interaction protocols are central design choices ([1, 2, 3]).

Diversity Metrics for Agent Systems

The measurement of behavioural diversity in multi-agent systems has been studied primarily in robotics. [9] introduces Hierarchic Social Entropy (HSE) as an information-theoretic measure of robot group diversity that captures both the number and relative size of behavioural clusters across all taxonomic scales. More recently, [10] propose System Neural Diversity (SND), which uses Wasserstein distance over multivariate robot-policy action distributions. SND requires all the agents to compute probabilities on the same action space and would not translate to heterogeneous LM agents with access to different tools. Our work adapts HSE to LM agent societies, replacing Euclidean inter-agent distance with cosine similarity over behavioural vectors, and is the first to apply such metrics to LM routing analysis. The ensemble learning literature has also studied diversity as a design objective [14], and quality-diversity optimisation [15] explicitly searches for a diverse set of high-performing solutions.

Specialisation in Multi-Agent Systems

A related line of work studies how individual actors come to occupy stable, distinct roles within a group — a prerequisite for routing to be meaningful over time. In multi-robot systems, specialisation is typically measured by the stability of roles rather than by task accuracy. [16] introduces response threshold models in which an agent's propensity to perform a task decreases each time it is performed by another agent; specialisation is said to emerge when an agent's threshold for a given task remains persistently low. [17] operationalises this differently via Collective Neuro-Evolution (CONE), defining a specialised agent as one that assumes a specific behavioural role for more than 50% of its operational lifetime.

Our routing robustness metric formalises this intuition for LM societies, measuring whether semantically-equivalent query variants are stably assigned to the same actor — the distributional consistency that these robotics accounts identify as the precondition for specialisation.

Robustness to Prompt Perturbations

A separate line of work studies the robustness of language models under input perturbations, including character-level noise, lexical substitutions, syntactic transformations, and paraphrases. A common finding is that meaning-preserving changes can cause large accuracy swings [11] across model sizes and instruction tuning. Prior work typically responds by training models to produce consistent outputs across clean and perturbed prompts ([18]), or by recommending that evaluations report performance ranges rather than point estimates ([11]). We study robustness at a different level: not whether the final answer is stable under perturbation, but whether the routing decision is.

By measuring whether perturbed queries are routed to the same actor as the original, we evaluate whether the routing policy induces stable task distributions over actors.

3. Preliminaries

Section Summary: The section sets out the core notation and framing for a “society” of AI actors—distinct models, prompts, or agents—together with a routing policy that assigns incoming queries to those actors. It shows how the authors measure behavioral differences among actors by building a matrix of their scores on a shared set of evaluation items, rather than by inspecting model weights. From this foundation the paper identifies two properties to study: how diverse the actors’ observed behaviors are, and how consistently a routing policy sends semantically similar queries to the same actor.

We introduce the notation used throughout the paper and state the two structural properties we study.

Let $\mathcal{R} = {r_1, r_2, \dots, r_N}$ be a society of $N$ actors, where each actor may correspond to a distinct model, prompt, tool-using agent, or role-specialised configuration. Actors may differ in architecture, scale, training data, or system prompt.

Let $\mathcal{Q} = {q_1, q_2, \dots, q_n}$ be a set of queries. A routing policy $\pi : \mathcal{Q} \to \mathcal{R}$ assigns each query to an actor; we write $a_i = \pi(q_i)$ for the actor assigned to query $q_i$. We denote by $\mathcal{Q}_k = {q_i \in \mathcal{Q} \mid a_i = r_k}$ the set of queries assigned to actor $r_k$.

Because actors may differ in architecture and parameterisation, we do not compare them in weight space. Instead, we evaluate each actor $r_k$ on an evaluation set $\mathcal{E} = {e_1, \dots, e_L}$, producing a behavioural vector,

$ \mathbf{b}_k = \bigl(s(r_k, e_1), ; s(r_k, e_2), ; \dots, ; s(r_k, e_L)\bigr) \in \mathbb{R}^{L} \nonumber\tag{1} $

where $s(r_k, e_l) \in [0, 1]$ is the score of actor $r_k$ on prompt $e_l$. Stacking these vectors gives a behavioural matrix $B \in \mathbb{R}^{N \times L}$, from which pairwise inter-actor distances are computed via cosine similarity (Section 4.2). Differences in parameter space do not necessarily correspond to differences in behavioural space ([12]); measuring diversity from $B$ rather than from model weights is therefore both more principled and more general, extending naturally to agent societies that share a model backbone but differ in system prompt or tool access. When a separate evaluation set is not available, $\mathcal{E}$ may be set equal to $\mathcal{Q}$.

Given a society $\mathcal{R}$ and a routing policy $\pi$, we study two properties that are orthogonal to task accuracy. Society diversity $\mathrm{HSE}(\mathcal{R})$ (Section 4) quantifies how behaviourally differentiated the actors in $\mathcal{R}$ are, as measured over the behavioural matrix $B$. This property is defined over the society alone and is independent of any routing policy. If $\mathrm{HSE}(\mathcal{R}) \approx 0$, all actors respond similarly and routing is vacuous regardless of the policy used. Routing robustness $\rho(\pi, \mathcal{Q})$ (Section 5) quantifies how stably $\pi$ assigns semantically-equivalent query variants to the same actor. This property is defined over the policy and depends on $\mathcal{R}$ only through the routing decisions $\pi$ induces. If $\rho(\pi, \mathcal{Q}) \approx 0$, actors receive inconsistent query distributions, which undermines the conditions under which specialisation can emerge.

Together, these two properties characterise when routing is meaningful: diversity ensures there is something non-trivial to route to, and robustness ensures that routing decisions satisfy the necessary condition for specialisation.

4. Society Diversity

Section Summary: The section introduces a scalar metric called hierarchic social entropy to quantify behavioral diversity within a society of actors, capturing both the number of distinct groups and the degree to which those groups differ in their behavioral profiles. It first defines simple social entropy via Shannon's formula applied to partitions of actors, then explains why this measure is insufficient because it ignores inter-group distances, and finally describes an adapted hierarchic version that uses hierarchical clustering on cosine-based distances between behavioral vectors. This approach, drawn from prior work on robot swarms, enables comparisons of societies and supports design choices such as routing without assuming diversity is always beneficial.

Given a society $\mathcal{R}$ and its behavioural matrix $B$ (Section 3), we want a single scalar capturing how behaviourally differentiated the actors are. A useful measure should reflect both the number of distinct behavioural groups and the degree to which those groups differ. Diversity is ubiquitous in natural systems ([19]), and ecologists have demonstrated the role of functional diversity in ecosystem survival ([20]). The challenge of identifying clusters of elements distributed in a continuous multidimensional space is exactly the problem faced by biologists constructing taxonomic systems, where each dimension represents a morphological trait distinguishing one organism from another. We adopt the same hierarchical view of entropy, replacing biological traits with the components of $\mathbf{b}_k$.

Specifically, we adapt Hierarchic Social Entropy (HSE; [9]), an information-theoretic measure originally proposed for robot swarms, to LM agent societies. We do not claim diversity is always desirable: for some tasks, homogeneous teams may outperform diverse ones. However, a quantitative metric enables controlled comparisons between societies and informs the design of diverse teams.

4.1 Simple Social Entropy

Simple social entropy applies Shannon's information entropy [21] to a multi-agent society to quantify behavioural diversity based on how actors are distributed among behavioural subsets.

Let $\mathcal{C} = {c_1, c_2, \dots, c_M}$ be a partition of $\mathcal{R}$ into $M$ homogeneous subsets, and let $p_i$ denote the proportion of actors belonging to subset $c_i$, with $\sum_{i=1}^{M} p_i = 1$. The simple social entropy of society $\mathcal{R}$ under partition $\mathcal{C}$ is:

$ H(\mathcal{R}) = -\sum_{i=1}^{M} p_i \log_2(p_i)\tag{2} $

This measure satisfies two basic properties: (i) homogeneity: $H(\mathcal{R}) = 0$ if and only if all actors fall into a single subset; (ii) maximum diversity: $H(\mathcal{R})$ is maximised when actors are distributed equally across all $M$ subsets ($p_i = 1/M$ for all $i$). Figure 1 illustrates simple social entropy on societies with varying cluster structure.

4.2 Inter-Agent Distance

To partition $\mathcal{R}$ into behavioural subsets, we require a pairwise distance over actors. Given the behavioural vectors $\mathbf{b}_k$ defined in Section 3, we measure the distance between actors $r_j$ and $r_k$ as one minus their cosine similarity:

$ d(r_j, r_k) = 1 - \frac{\mathbf{b}_j \cdot \mathbf{b}_k} {|\mathbf{b}_j|, |\mathbf{b}_k|}\tag{3} $

We prefer cosine similarity over the Euclidean distance used in the original HSE formulation ([9]) because behavioural vectors may differ substantially in magnitude when agents vary in overall accuracy, and we wish to capture profile similarity rather than absolute performance level (Appendix E). We prefer it over the Wasserstein distance used in SND ([10]) because that metric is defined over full action distributions, which are not directly available for language model outputs.

4.3 Hierarchic Social Entropy

Simple social entropy Equation (2) captures two aspects of diversity: the number of behavioural groups and the distribution of actors across them. It does not, however, capture the degree of difference between groups. Figure 2 illustrates the consequence: two societies each containing three identical actors and one outlier receive the same score $H = 0.811$, regardless of whether the outlier is close to or far from the main cluster ($d_1 \ll d_2$, yet $H$ is identical). This limitation matters for routing. Suppose a router assigns queries to an outlier actor that barely differs from the rest of the society. Simple entropy would record non-zero diversity, yet the routing decision is nearly vacuous: the outlier provides almost no complementary capability. A useful diversity measure should be sensitive to the degree of behavioural separation between groups, not just to whether separation exists.

To obtain a single society-level score, we aggregate pairwise inter-actor distances (defined in Section 4.2). [10] aggregate by taking the mean over all pairwise distances. This is computationally simple but conflates societies with very different cluster structures: a society with two tight, well-separated clusters and a society with uniformly scattered actors can yield the same mean distance, yet the former is far more useful for routing, since actors fall into stable, distinguishable groups. [9] instead cluster actors at every possible distance threshold and measure entropy at each scale. We adopt this approach for the same reason: it is sensitive to the hierarchic structure of the behavioural space, not just its average extent.

Specifically, we use hierarchical clustering with single linkage (nearest-point algorithm), which merges two clusters as soon as their closest members fall within threshold $h$. This is appropriate for our setting, where even a single behaviourally distinct actor justifies a separate cluster.[^1] As $h$ varies, the partition $\mathcal{C}$ changes, so simple social entropy becomes a function of both the society and $h$, written $H(\mathcal{R}, h)$. We call $h$ the taxonomic level, borrowing the term from [9], who in turn draws on biological taxonomy: just as a biologist can classify organisms at the species level or the genus level depending on how finely they wish to distinguish, $h$ controls the resolution at which we observe the society. At $h = 0$, actors merge only if their distance is exactly zero; behaviourally distinct actors each form a singleton cluster, giving $H(\mathcal{R}, 0) = \log_2 M$ where $M \leq N$ is the number of distinct behavioural profiles. As $h$ increases, similar actors merge and the number of clusters falls, until at $h \to \infty$ all actors collapse into one group with $H(\mathcal{R}, h) = 0$.

[^1]: The original HSE formulation uses Euclidean distance and centroid linkage rather than cosine distance and single linkage; we compare both in Appendix E.

Hierarchic Social Entropy integrates simple entropy across all taxonomic levels:

$ \mathrm{HSE}(\mathcal{R}) = \int_{0}^{\infty} H(\mathcal{R}, h), dh\tag{4} $

We define $d(r_j, r_k) = 1$ (maximum distance) when either $|\mathbf{b}_j| = 0$ or $|\mathbf{b}_k| = 0$, i.e., when an actor fails every evaluation prompt. HSE is a continuous ratio measure with an absolute zero (when all actors are identical), enabling statements of the form "Society A is twice as diverse as Society $B''$ and total orderings across societies of different sizes when normalised by the maximum achievable HSE for that society size.

4.4 Validating HSE

::: {caption="Table 1: HSE sanity checks. (a) Identical models produce zero diversity. (b) Orthogonal models achieve maximum diversity (normalised HSE = 1.0 regardless of society size). p1–p3 represent different prompts."}

:::

We construct minimal societies that isolate specific properties of HSE, confirming that the metric behaves as expected before applying it at scale (see Section 7). Table 1 verifies that when all actors produce identical outputs (a), the society is maximally homogeneous: HSE is zero and routing is vacuous, regardless of how many actors are present. When actors are fully orthogonal, each succeeding on exactly the prompts the others fail (b), HSE is maximised and normalised HSE equals 1.0. Crucially, the normalised value is invariant to society size, enabling fair comparisons across societies of different scales.

Table 2 verifies that adding redundant information does not artificially inflate HSE. Starting from a base society of three behaviourally distinct actors (case a), we extend it in two ways: adding a fourth actor whose behavioural vector is identical to actor C (case b), and adding a fourth evaluation prompt that is a copy of p3 (case c). In both cases HSE decreases slightly rather than increasing, confirming that the metric is not fooled by redundancy. This property matters in practice: a routing system should not appear more diverse simply because actors are duplicated or because the evaluation set contains repeated queries. Finally, Figure 3 empirically confirms monotonicity: HSE increases as behaviourally distinct actors are added.

::: {caption="Table 2: HSE on three related cases sharing a common base matrix (unshaded 3×3 block, case a). The shaded column extends the base with a duplicate model D = C (case b); the shaded row extends it with a duplicate prompt p4 = p3 (case c). Adding duplicates does not increase diversity."}

:::

5. Robustness of Routing Policy

Section Summary: The section argues that actor diversity alone cannot produce stable specialization unless routing policies consistently send similar queries to the same actor over time. It therefore defines a robustness metric that measures how often a router assigns surface-level or meaning-preserving variants of a query—ranging from typos to full paraphrases and added irrelevant text—to the same actor as the original query. The metric can be computed in aggregate or per actor, revealing where a policy becomes unstable under progressively larger changes in wording while assuming that semantically equivalent inputs should receive equivalent assignments.

Society diversity (Section 4) characterises whether a society contains actors with sufficiently differentiated capabilities to make routing non-trivial. But diversity alone is not sufficient: as the specialisation literature makes clear (Section 2), actors can only develop stable roles if they reliably receive the same type of queries over time.

We now introduce a metric that captures this second structural property directly.

The central intuition is that stable query assignment is a necessary condition for specialisation to be possible: a router that assigns query $q_i$ to actor $r_k$ but assigns a surface-form variant to actor $r_j \neq r_k$ violates this condition, regardless of what learning dynamics operate downstream. We do not claim that stable routing produces specialisation — whether actors develop specialised competence depends on factors outside the scope of this work — but unstable routing makes it impossible by construction. We therefore measure routing robustness as the degree to which surface-form variants of a query are assigned to the same actor.

5.1 Perturbation Taxonomy

The robustness metric $\rho$ is defined relative to a perturbation regime: the same router may be stable under surface-level noise yet unstable under semantic reformulation. To support both aggregate and fine-grained analysis, we define five perturbation classes of increasing semantic distance from the original query, summarised in Table 3.

: Table 3: Perturbation taxonomy, ordered by increasing semantic distance from the original query.

Level	Class	Example
1	Character-level	Typos, spacing, punctuation
2	Word-level	Synonym substitution
3	Syntax-level	Passive voice, clause reordering
4	Paraphrase	Full rewrite, same meaning
5	Rambling	Added irrelevant context

Perturbations at levels 1–3 preserve surface proximity to the original query; levels 4–5 test whether the router is sensitive to meaning-preserving reformulations that substantially alter wording or add noise. Full methodology and prompt templates used to generate each class of perturbations are provided in Appendix B.1. By computing $\rho$ separately for each perturbation class, one can characterise where a routing policy breaks down: a router that degrades only at level 4–5 is more robust in practice than one that degrades at level 2, even if their aggregate $\rho$ scores are similar.

5.2 Routing Robustness Metric

Given a routing policy $\pi$ and a query $q_i \in \mathcal{Q}$, recall that $a_i = \pi(q_i)$ denotes the actor assigned to the original query (Section 3). We generate $p$ perturbations $q_i^1, q_i^2, \ldots, q_i^p$ of $q_i$ (see Section 5.1 for the perturbation taxonomy). The per-query robustness is the fraction of perturbations routed to the same actor as the original:

$ \rho_i = \frac{1}{p} \sum_{j=1}^{p} \mathbf{1}!\left[\pi(q_i^j) = a_i\right]\tag{5} $

where $\mathbf{1}[\cdot]$ is the indicator function. $\rho_i \in [0, 1]$, with $\rho_i = 1$ if all perturbations are routed to the same actor, and $\rho_i = 0$ if none are.

Dataset-level robustness is obtained by averaging over all $n = |\mathcal{Q}|$ queries:

$ \rho(\pi, \mathcal{Q}) = \frac{1}{n} \sum_{i=1}^{n} \rho_i\tag{6} $

This metric rests on the assumption that meaning-preserving perturbations do not change which actor is most accurate for a given query: if $q_i$ and $q_i^j$ are semantically equivalent, the optimal assignment should be the same for both. Under this assumption, any change in routing decision reflects instability in the policy rather than a legitimate response to a change in query difficulty or actor suitability.

Two boundary cases anchor the metric: a fixed router that always assigns the same actor gives $\rho_i = 1$ for all $i$ and $\rho = 1$, while a uniform random router gives $\mathbb{E}[\rho] = 1/N$. Beyond the dataset-level aggregate, we also compute a per-actor score over the query set $\mathcal{Q}_k$ assigned to actor $r_k$:

$ \rho^{(k)}(\pi, \mathcal{Q}) = \frac{1}{|\mathcal{Q}k|} \sum{q_i \in \mathcal{Q}_k} \rho_i\tag{7} $

If no queries are assigned to actor $r_k$ under policy $\pi$ (i.e., $|\mathcal{Q}_k| = 0$), we define $\rho^{(k)}(\pi, \mathcal{Q}) = 0$. A low $\rho^{(k)}$ indicates that queries nominally assigned to actor $r_k$ are unstably routed under perturbation, violating the necessary condition for specialisation to emerge for that actor's role.

6. Experimental Setup

Section Summary: The experimental setup evaluates model routing across two benchmarks, EmbedLLM and RouterBench, which supply performance data for large pools of language models on reasoning and knowledge tasks. These pools, called societies, include both the original models and several synthetic groups engineered to have varying degrees of specialized expertise, along with selected high-diversity subsets. Five routing methods are tested, ranging from random assignment to nearest-neighbor lookup and LLM-based selection, with robustness measured by applying five graduated perturbations to each query and tracking how well each method maintains performance.

Datasets

We evaluate on two routing benchmarks. EmbedLLM ([13]) provides performance scores across 112 models on 10 reasoning and knowledge benchmarks (MMLU, GSM8K, GPQA, ASDiv, and others), comprising 36, 054 questions in total. RouterBench ([7]) is a complementary routing benchmark with performance scores across 11 models on 36, 497 queries drawn from 8 datasets (see Appendix A for details and train, validation, and test splits).

Societies

The benchmarks described above constitute the default societies. On EmbedLLM, we compare the default society (HSE=1.62) against three synthetic societies in which models are replaced by synthetic experts with binary performance indicators encoding domain-specific expertise:

• RD (Reasoning Depth): five non-overlapping experts covering five tiers of cognitive complexity, ranging from direct lookup to complex multi-step reasoning (HSE=2.32).
• SA (Subject Area): 15 non-overlapping experts each specialising in a distinct academic or practical domain, such as advanced mathematics, computer science, medicine, and commonsense reasoning (HSE=3.91).
• RD+SA: the combined pool of 20 RD and SA experts with partially overlapping expertise (HSE=3.19).

Expert categorisation heuristics are detailed in Appendix A.3. Oracle accuracy over each synthetic society is 1.0 by construction, since each query falls within exactly one expert's domain. We define a max-HSE subset for each benchmark: the subset of the full society that maximises normalised HSE at a fixed society size, selected greedily; this subset serves as a high-diversity reference within real-world model pools. On RouterBench we compare the full society against its max-HSE subset.

Routing Policies

We evaluate five routing policies: random assigns each query to a uniformly sampled actor; prompted prompts an LLM with natural-language descriptions of actor roles and few-shot examples to select the most appropriate one (see Appendix C); KNN- $k$ ($k \in {1, 3, 10}$) retrieves the $k$ nearest neighbours of the query in embedding space and assigns the actor that performed best on those neighbours (Appendix D).

Perturbations

Routing robustness (Section 5.2) is measured over the five-level perturbation taxonomy of Section 5.1. For each query we generate one perturbation per level, yielding $p = 5$ variants (Appendix B.1); we report dataset-level robustness $\rho(\pi, \mathcal{Q})$ Equation (6) and per-level breakdowns.

7. Results and Analysis

Section Summary: Specialist societies built from deliberately chosen models prove far more behaviourally diverse than the default pool of real-world models, yet all societies show sharply diminishing returns once a handful of agents are included. KNN routers exploit this diversity to raise accuracy on clean queries, but they become brittle on slight rephrasings, whereas prompted routers trade a little clean accuracy for steadily higher robustness that improves along with diversity. Selecting the most diverse subsets further strengthens prompted routing on both benchmarks and rescues KNN robustness on one, while overall robustness remains largely unchanged across varying degrees of query perturbation.

Specialist societies are substantially more diverse.

Figure 3 shows normalised HSE as a function of society size on EmbedLLM. At any fixed size, purpose-designed specialist societies achieve higher HSE than the default real-world model pool. The SA society (HSE = 3.91) is more than twice as diverse as the default (HSE = 1.62), and the RD+SA combined pool (HSE = 3.19) similarly outpaces it despite containing many fewer total actors. This indicates that a large number of real-world models occupy similar regions of behavioural space: size alone does not imply diversity.

Across all societies, HSE exhibits strong diminishing returns: the marginal gain from adding an actor falls sharply after the first few. For the default society, the curve plateaus after approximately nine agents. Synthetic specialist societies plateau even faster, at around four to five agents for RD and SA, because their behavioural profiles are maximally orthogonal by construction. Small, carefully curated societies can recover most of the diversity available in a much larger pool, and collecting additional models offers little incremental benefit. HSE can also be combined with task accuracy as a joint selection criterion: Appendix F reports results for subsets selected by maximising HSE jointly with task accuracy, showing that the joint criterion leads to better routing while preserving diversity.

KNN routers gain accuracy from specialist societies but collapse in robustness.

Figure 4(a) shows task accuracy rising sharply and monotonically for KNN routers as HSE increases. However, perturbed-query accuracy collapses at every specialist society, falling to near-random levels. Panel (b) confirms that robustness $\rho$ remains low across the entire HSE range despite high clean-query accuracy. The sharp behavioural boundaries that make specialist societies easy to route correctly on clean queries make them equally easy to route incorrectly when a surface-level reformulation shifts the query embedding across a domain boundary.

The prompted router shows the reverse pattern. Its clean accuracy on SA is lower than KNN-10, but perturbed accuracy remains close, a gap of less than five points. Crucially, robustness rises monotonically with HSE, tracking society diversity in a way no KNN router does. Prompted routing latches onto the semantic intent of the query rather than its embedding position, making it naturally stable under meaning-preserving reformulations.

Max-HSE subsets improve prompted robustness, but the effect depends on pool structure.

Figure 5 compares the full society against the max-HSE subset on EmbedLLM and RouterBench. The prompted router improves consistently, roughly doubling its robustness on EmbedLLM and improving more modestly on RouterBench. The KNN routers tell a more complex story. On RouterBench, switching to the max-HSE subset dramatically rescues KNN robustness across all values of $k$. On EmbedLLM, however, KNN-1 robustness actually decreases while KNN-3 and KNN-10 improve only marginally. This contrast reflects the nature of the max-HSE subset in each case: the RouterBench subset selects from models with smooth, graded behavioural profiles, so higher diversity sharpens actor distinctions without producing hard domain boundaries; the EmbedLLM pool, even at maximum HSE, has less structured behavioural separation, leaving nearest-neighbour routing sensitive to small surface variations in the query.

Robustness is roughly flat across perturbation levels.

Figure 6 shows per-level robustness on RouterBench. Every router produces approximately the same $\rho$ at all five levels: the prompted router ranges from 0.589 (paraphrase) to 0.617 (syntax) on the full society, and from 0.663 (paraphrase) to 0.744 (word) on the max-HSE subset. No router shows a consistent decline as perturbations become semantically deeper. Routing policies are no more sensitive to a full paraphrase than to a character-level typo, implying that routing decisions are not anchored to surface features that higher-level perturbations would selectively disrupt. The only hint of level-sensitivity is the prompted router's paraphrase dip on the max-HSE subset (0.663 vs. $\approx!0.73$ elsewhere), suggesting that full rewrites may destabilise an LLM-based policy when actor distinctions are sharper.

8. Conclusion

Section Summary: Researchers argue that routing systems for groups of language models need to be judged not just on accuracy but also on how diverse the models' behaviors are and how consistently the router assigns similar queries to the same model. They adapted an entropy measure to quantify behavioral variety and created a test that checks whether small changes in wording shift the routing decisions. Their experiments show that only a handful of models deliver most of the useful diversity, while simple nearest-neighbor routers prove accurate yet brittle and prompted routers prove steadier but less precise.

We have argued that routing in language-model societies should be evaluated along two structural dimensions beyond task accuracy: behavioural diversity and routing stability under surface-form query variants. For diversity, we adapted Hierarchic Social Entropy to language-model societies, using cosine similarity over behavioural vectors as the inter-actor distance. For robustness, we introduced a perturbation-based metric measuring the fraction of surface-form variants assigned to the same actor.

Applied to EmbedLLM and RouterBench, our analysis finds that HSE exhibits strong diminishing returns: fewer than ten agents capture most available diversity, providing a practical heuristic for society design. In addition, robustness and accuracy trade off sharply: KNN routers achieve high accuracy on specialist societies but collapse under perturbation, whereas prompted routers are less accurate but more stable across perturbation types and society configurations.

Together, HSE and routing robustness ask not only whether a router performs well, but whether routing is meaningful: whether the society is sufficiently differentiated to justify routing decisions, and whether the policy satisfies the necessary condition for specialisation.

9. Limitations

Section Summary: The limitations section notes that HSE measurements rely on a fixed evaluation set and a greedy selection process that may not match real query patterns or find the best possible groupings, potentially skewing diversity estimates. It also highlights that the study’s synthetic societies use artificial binary success indicators, which create sharper distinctions than would likely appear with real, more overlapping agent capabilities. Finally, the robustness metric only examines single-turn query assignments and does not track whether agents actually gain lasting specialized skills or how issues might spread across multi-step tasks.

Measurement assumptions

HSE is computed over a fixed evaluation set $\mathcal{E}$; if this set is not representative of the actual query distribution, behavioural vectors may not reflect deployment behaviour and diversity estimates could be misleading. The max-HSE subset is selected greedily, which does not guarantee the globally optimal subset at a given size; the true optimum requires an exponential search. Our synthetic societies further use binary performance indicators that produce maximally sharp behavioural boundaries and oracle accuracy of 1.0 by construction; real specialist agents will exhibit softer, overlapping profiles, and the KNN brittleness we observe may be attenuated in practice.

Scope of the robustness metric

Routing robustness is measured at the assignment level over single-turn queries and does not capture whether actors actually develop specialised competence as a result of stable query distributions. Extending the metric to multi-step agentic settings, where perturbations propagate through state rather than through a single query, is a natural next step, as is using HSE as a training signal to bootstrap diverse societies from a homogeneous initialisation.

Appendix

Section Summary: The appendix describes two primary benchmarks used to study model routing. EmbedLLM supplies a large matrix of correct/incorrect answers from 112 open-source models across questions drawn from ten standard tests, while RouterBench provides responses, quality scores, and costs from eleven models on eight datasets. It also explains the construction of synthetic “societies” in which specialized models are created by clustering evaluation items into distinct subject areas, allowing controlled experiments on routing performance.

A. Description of Router Benchmarks

A.1 EmbedLLM

EmbedLLM ([13]) is a model routing and correctness-forecasting benchmark built around a matrix of binary performance labels collected from 112 open-source language models spanning a wide range of sizes and specialisations (general-purpose, coding, biomedical, physics). Each model was evaluated on 36, 054 questions drawn from the test sets of ten established benchmarks, summarised in Table 4. For each model–question pair the label records whether the model answered correctly (1) or not (0), yielding a correctness matrix $Y \in {0, 1}^{112 \times 36{,}054}$. Some questions were duplicates or near-duplicates which differed only by the order in which the multiple choice answers were presented, so we filtered out about 3k prompts. The remaining questions were split 80%/10%/10% into train, validation, and test sets (approximately 26K/3.6K/3.6K questions).

In our work, we treat each model as an actor $r_k \in \mathcal{R}$ and each row of $Y$ as the actor's behavioural vector $\mathbf{b}_k$ (Section 3). The full set of 112 models constitutes the default society; the max-HSE subset is the 9-model subset selected greedily to maximise normalised HSE (Section 6).

\begin{tabularx}{\lcccccccc}{@lX@}
\toprule
\textbf{Benchmark} & \textbf{Domain} \\
\midrule
MMLU ([22]) & General knowledge (57 subjects) \\
TruthfulQA ([23]) & Truthfulness / factuality \\
SocialQA ([24]) & Social commonsense reasoning \\
PIQA ([25]) & Physical commonsense reasoning \\
MedMCQA ([26]) & Medical knowledge \\ 
MathQA ([27]) & Mathematical word problems \\ 
LogiQA ([28]) & Logical reasoning \\
GSM8K ([29]) & Grade-school math reasoning \\
GPQA ([30]) & Graduate-level science (hard) \\ 
ASDiV ([31]) & Arithmetic word problems \\  
\bottomrule
\end{tabularx}

A.2 RouterBench

RouterBench ([7]) is a comprehensive benchmark for evaluating multi-LLM routing systems, designed to cover a broad spectrum of tasks and domains. The dataset contains 405, 467 pre-generated inference samples from 11 LLMs — six open-source (Llama-70B-chat, Mixtral-8x7B, Yi-34B-chat, Code Llama-34B, Mistral-7B-chat, WizardLM-13B) and five proprietary (GPT-4, GPT-3.5-turbo, Claude-instant-v1, Claude-v1, Claude-v2) — evaluated across 8 datasets spanning 64 tasks (Table 5).

Each sample records the model response alongside a quality score and an inference cost in dollars, enabling joint evaluation of task performance and economic cost. For routing experiments, the dataset is partitioned 80%/10%/10% into train, validation, and test sets (approximately 29K/3.6K/3.6K prompts).

In our work, we treat each of the 11 models as an actor $r_k \in \mathcal{R}$, using binary correctness labels as the entries of the behavioural matrix $B$ (Section 3). The full set of 11 models constitutes the default society; the max-HSE subset is the 4-model subset that maximises normalised HSE, selected greedily (Section 6).

\begin{tabularx}{\lcccccccc}{@lX@}
\toprule
\textbf{Benchmark} & \textbf{Domain} \\
\midrule
HellaSwag ([32]) & Commonsense Reasoning \\
Winogrande ([33]) & Commonsense Reasoning \\
ARC Challenge ([34]) & Commonsense Reasoning \\
 MMLU ([22]) & Knowledge Understanding \\
MT-Bench ([35]) & Conversation \\
 GSM8K ([29]) & Math \\
MBPP ([36]) & 1 Coding \\

\bottomrule
\end{tabularx}

A.3 Synthetic Societies

Subject-Area Clustering (SA)

To construct the SA synthetic society we assign each of the 67 evaluation items in EmbedLLM, comprising 57 MMLU subjects and 10 additional benchmarks (GSM8K, MathQA, ASDiv, LogiQA, GPQA, MedMCQA, TruthfulQA, SocialQA, PIQA), to one of 15 mutually exclusive subject-area clusters. The assignment criterion is academic domain: items are grouped by what they are about, independently of how much reasoning they require. The resulting clusters are shown in Table 6.

: Table 6: Subject-area clustering of the 67 EmbedLLM evaluation items into 15 groups. The MMLU subjects column lists the 57 individual MMLU subjects assigned to each cluster; the EmbedLLM column lists the 10 additional benchmarks. Each synthetic SA expert scores 1 on every item in its cluster and 0 on all others. HS = High School; CS = Computer Science.

ID	Cluster	MMLU subjects	EmbedLLM
C01	Elementary & school arithmetic	Elementary mathematics, HS statistics	GSM8K, ASDiv
C02	Advanced & university mathematics	Abstract algebra, college mathematics, HS mathematics, econometrics	MathQA
C03	Logic, argumentation & formal reasoning	Formal logic, logical fallacies	LogiQA
C04	Physics & astronomy	Astronomy, college physics, conceptual physics, HS physics	—
C05	Chemistry & earth sciences	College chemistry, HS chemistry	GPQA
C06	Biology, genetics & virology	College biology, HS biology, medical genetics, virology	—
C07	Clinical medicine & healthcare	Anatomy, clinical knowledge, college medicine, human aging, human sexuality, nutrition, pre-medical, professional medicine	MedMCQA
C08	Computer science, AI & cybersecurity	College CS, computer security, electrical engineering, HS CS, machine learning	—
C09	Economics, business & accounting	HS macroeconomics, HS microeconomics, management, marketing, professional accounting, public relations	—
C10	Law & legal reasoning	Business ethics, international law, jurisprudence, pre-law, professional law	—
C11	Ethics, philosophy & religion	Moral disputes, moral scenarios, philosophy, world religions	—
C12	Psychology & behavioural sciences	HS psychology, professional psychology, sociology	SocialQA, TruthfulQA
C13	Politics, international relations & security	HS government & politics, security studies, US foreign policy	—
C14	History, geography & world knowledge	Global facts, HS European history, HS geography, HS US history, HS world history, miscellaneous	—
C15	Commonsense & physical intuition	—	PIQA

Each synthetic SA agent is assigned a binary performance vector, scoring 1 on every item in its cluster and 0 on all others, modelling a specialist with perfect but narrowly scoped expertise. Several borderline assignments are worth noting. Econometrics is placed in Advanced Mathematics rather than Economics because its questions are dominated by statistical modelling and regression rather than economic theory. GPQA is placed in Chemistry & Earth Sciences because its hardest and most distinctive questions are in graduate-level chemistry and physical sciences. TruthfulQA is placed in Psychology & Behavioural Sciences because its core demandm resisting socially transmitted false beliefs, is an epistemic and psychological phenomenon rather than a factual one. Business Ethics is placed in Law & Legal Reasoning because its MMLU questions predominantly test corporate governance and fiduciary duty rather than philosophical ethics.

Reasoning-Depth Clustering (RD)

To construct the RD synthetic society we assign the same 67 EmbedLLM items to one of five reasoning-depth tiers that form a strictly ordered scale from surface recall to adversarial expert reasoning. The assignment criterion is the dominant cognitive operation required to answer correctly, independent of subject matter.

As with SA, each RD agent scores 1 on every item in its tier and 0 elsewhere. Key borderline decisions include: TruthfulQA is assigned to R2 rather than R3 because each individual question requires a single evaluative judgment, not a reasoning chain; LogiQA is assigned to R4 rather than R3 because its civil-service puzzles require applying formal inference rules rather than multi-step domain reasoning; ASDiv is assigned to R4 rather than R1 because its design goal is robustness across diverse arithmetic structures, probing rule application rather than recall; and GPQA is assigned to R5 because it is explicitly designed so that non-experts perform near chance and domain experts reach only around 65%, with distractors crafted to exploit common expert heuristics.

B. Query perturbations

B.1 Perturbation generation

We generate the query perturbations by prompting an LM, using Gemini 3.1 Flash-Lite [37] as backbone model. This section documents the five zero-shot prompts used to generate perturbations of each query in the evaluation sets of EmbedLLM and RouterBench (Section 5.1). Each prompt instructs the model to produce 10 rewrites of an input query prompt according to specific perturbation instructions for each of the five levels: character, word, syntax, paraphrase, and rambling. We request more perturbations than needed since occasionally the model produces duplicates variants. We then parse the outputs and select 5 unique variants per perturbation category.

B.2 Query perturbation examples

This section presents some examples of the generated prompt perturbations.

~~~~ {caption="field header"}

Original Prompt

How to reduce the pain of a sore throat with a natural substance?

Perturbation Variants

~~~ {caption="EmbedLLM Perturbation Example 2"}

~~~~ {caption="field header"}

Original Prompt

How can I get something sweet even when I am not eating sugar?

Perturbation Variants

~~~ {caption="RouterBench Perturbation Example 1"}

~~~~ {caption="field header"}

Original Prompt

Write a function to find the n'th lucas number.

Perturbation Variants

~~~ {caption="RouterBench Perturbation Example 2"}

~~~~ {caption="field header"}

Original Prompt

Write a function to check if a dictionary is empty

Perturbation Variants

### C. Prompted Router Policy

For the prompted router policy, we use a few-shot prompt template with Gemini 3.1 Flash-Lite [37] as LM backbone. The prompt asks a language model to select the best-suited actor for a given query, conditioned on (i) a natural-language description of each available actor, and (ii) a set of few-shot exemplars drawn from the training split.

Appendix C.1 presents the template itself; Appendix C.2 describes how few-shot exemplars are selected; and Appendix C.3 show how the template is instantiated for the EmbedLLM and RouterBench datasets.

#### C.1 Prompt Template

![](https://ittowtnkqtyixxjxrhou.supabase.co/storage/v1/object/public/public-images/vvbma4jp/complex_fig_a2163c3b25fa.png)

#### C.2 Few-Shot Exemplar Selection

For each actor, $k$ exemplars are randomly sampled from training prompts where that actor is the clear winner, i.e., its score exceeds all other actors by at least a margin $\delta$. We use $k{=}4$ and $\delta{=}0.4$. In the rare instances in which there are less than $k$ prompts that satisfy this criteria, we fall back to sampling from training prompts where that actor achieved the highest score among all actors in the society (possibly tied with other actors).

Exemplars are formatted as below and shuffled together across all specialists:

~~~ {caption="output box"}

PROMPT: `{exemplar_query}`

Best model for this prompt: `{actor}`

C.3 Instantiation

For EmbedLLM and RouterBench we list all the names of the LLMs available in the dataset, e.g., huggingfaceh4_zephyr_7b_beta, in the actors_description field.

For the synthetic societies, each actor corresponds to a knowledge domain (subject area), a reasoning depth, or both, so we inject the descriptions below into the actors_description placeholder.

- `elementary_math`: a specialist in foundational numerical operations and word problems at primary/middle-school level
- `advanced_math`: a specialist in university-level pure and applied mathematics requiring symbolic manipulation, proof, or multi-domain quantitative reasoning
- `logic`: a specialist in the application of formal logical rules, identification of valid/invalid argument structures, and detection of informal fallacies
- `physics`: a specialist in principles of mechanics, electromagnetism, thermodynamics, quantum physics, and astrophysics at high-school to graduate level
- `chemistry`: a specialist in chemical principles from stoichiometry and bonding to organic reactions and physical chemistry, plus graduate-level natural science breadth
- `biology`: a specialist in life sciences spanning molecular biology, genetics, evolutionary biology, ecology, and virology at school to university level
- `medicine`: a specialist in clinical diagnosis, patient management, pharmacology, and healthcare practice at the level of medical school and professional licensing
- `computer_science`: a specialist in algorithms, data structures, machine learning models, computer systems, and security—theoretical and applied CS at school to graduate level
- `economics`: a specialist in micro/macroeconomic theory, business management, marketing, public relations, and professional accounting
- `law`: a specialist in legal theory, jurisprudence, international and domestic law at the level of law school entry and professional bar examination
- `philosophy`: a specialist in normative ethics, applied moral reasoning, philosophy of mind and knowledge, and comparative religion
- `psychology`: a specialist in cognitive, developmental, clinical, and social psychology, plus professional assessment and therapy practice
- `politics`: a specialist in political institutions, government, foreign policy, international security, and strategic studies
- `history`: a specialist in historical events and movements (European, US, world), human and physical geography, and broad cross-national factual knowledge
- `commonsense`: a specialist in implicit everyday knowledge about how the physical world works and how people behave

- `direct_lookup`: a model whose answers are stored facts retrievable in one step, with no inference, calculation, or reasoning chain needed
- `concept_application`: a model that applies a concept, rule, or definition to a specific case in one inferential step, requiring understanding beyond rote recall but no chaining
- `multi_step`: a model that reaches the answer by composing two or more distinct reasoning steps, each depending on the output of the previous
- `formal_symbolic`: a model that derives the answer by applying a formal system (logic, algebra, proof) according to explicit syntactic rules
- `adversarial`: a model tested on questions specifically designed to defeat shallow heuristics, requiring deep domain expertise, resistance to plausible distractors, and often meta-cognitive awareness

D. KNN Router Policy

The KNN-based router computes an embedding $e(q)$ of the query using Gemini Embedding 2 as model backbone ([38]) and retrieves the $k$ nearest neighbours $\mathcal{N}_k(x)$ from the training set using cosine similarity in the embedding space. It then assigns the actor with highest score over the $k$ nearest neighbours.

E. Effect of Distance Metric and Linkage on HSE

Figure 7 reports normalised HSE as a function of society size across all twelve combinations of distance metric (cosine, correlation, euclidean, manhattan) and linkage method (single, complete, average), using greedy agent selection on EmbedLLM. The qualitative pattern is consistent across configurations: specialist societies plateau rapidly at high HSE while the default real-world pool plateaus at a substantially lower level, confirming that the finding reported in Section 7 does not depend on the choice of distance or linkage. Cosine distance with single linkage, used throughout the main paper, produces the most conservative estimates of diversity for the default society.

F. Effect of Model Selection Criteria

\begin{tabularx}{\lcccccccc}{l r *6{R}}
\toprule
 {} & $n$ & HSE & rand. & KNN1 & KNN3 & KNN10 & orac.\\
\midrule
 default & 112 & 0.24 & 0.43 & 0.52 & 0.57 & 0.65 & 0.97 \\
 \midrule
\multirow{3}{*}{HSE} & 9 & 0.53 & 0.30 & 0.34 & 0.40 & 0.46 & 0.87 \\
  {} & 12 & 0.49 & 0.30 & 0.35 & 0.41 & 0.46 & 0.89 \\
  {} & 25 & 0.41 & 0.35 & 0.48 & 0.50 & 0.52 & 0.94 \\
 \midrule
\multirow{3}{*}{HSE + acc.} & 9 & 0.48 & 0.33 & 0.52 & 0.59 & 0.65 & 0.92 \\
  {} & 12 & 0.47 & 0.34 & 0.52 & 0.58 & 0.65 & 0.93 \\
  {} & 25 & 0.41 & 0.35 & 0.46 & 0.55 & 0.63 & 0.96 \\
\bottomrule
\end{tabularx}

Table 7 reports task accuracy and society diversity for subsets of the EmbedLLM society selected under two criteria: maximising HSE alone, and maximising HSE jointly with task accuracy. The results reveal a diversity-accuracy tradeoff. Subsets selected purely on HSE achieve higher normalised HSE at every size but lower routing task accuracy. Adding task accuracy to the selection criterion recovers routing performance — at $n{=9}$, KNN-10 matches the full 112-model society exactly (0.65) and oracle reaches 0.92 — but at the cost of lower HSE (0.48 vs. 0.53). The HSE + acc. subset at $n{=9}$ achieves competitive routing accuracy using fewer that 10% of the available models, providing a practical coreset that balances diversity and performance.

References

Section Summary: This references section compiles a list of academic papers and technical reports, mostly from recent AI and machine learning conferences. The works focus on multi-agent systems powered by large language models, methods for routing and combining models efficiently, ways to measure diversity in groups or algorithms, and benchmarks for evaluating model performance and robustness. Many draw connections to biology, robotics, and information theory to inform the design of more capable AI collectives.

[1] Wu et al. (2024). AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversations. In Proceedings of the First Conference on Language Modeling. https://openreview.net/forum?id=BAakY1hNKS.

[2] Hong et al. (2024). MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=VtmBAGCN7o.

[3] Li et al. (2023). CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society. In Advances in Neural Information Processing Systems. https://openreview.net/forum?id=3IyL2XWDkG.

[4] Yue et al. (2025). MasRouter: Learning to Route LLMs for Multi-Agent Systems. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 15549–15572. doi:10.18653/v1/2025.acl-long.757. https://aclanthology.org/2025.acl-long.757/.

[5] Chen et al. (2024). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. Transactions on Machine Learning Research. https://openreview.net/forum?id=cSimKw5p6R.

[6] Isaac Ong et al. (2025). RouteLLM: Learning to Route LLMs from Preference Data. In The Thirteenth International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=8sSqNntaMr.

[7] Hu et al. (2024). RouterBench: A Benchmark for Multi-LLM Routing System. In ICML 2024 Workshop on Agentic Markets. https://openreview.net/forum?id=IVXmV8Uxwh.

[8] Dongfu Jiang et al. (2023). LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL). pp. 14165–14178. https://aclanthology.org/2023.acl-long.792.

[9] Balch, Tucker (2000). Hierarchic social entropy: An information theoretic measure of robot group diversity. Autonomous robots. 8(3). pp. 209–238.

[10] Bettini et al. (2025). System neural diversity: Measuring behavioral heterogeneity in multi-agent learning. Journal of Machine Learning Research. 26(163). pp. 1–27.

[11] Melanie Sclar et al. (2024). Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I Learned to Start Worrying about Prompt Formatting. In The Twelfth International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2310.11324.

[12] Stanley et al. (2019). Designing neural networks through neuroevolution. Nature Machine Intelligence. 1(1). pp. 24–35.

[13] Zhuang et al. (2025). EmbedLLM: Learning Compact Representations of Large Language Models. In The Thirteenth International Conference on Learning Representations. https://arxiv.org/abs/2410.02223.

[14] Krogh, Anders and Vedelsby, Jesper (1994). Neural Network Ensembles, Cross Validation, and Active Learning. In Advances in Neural Information Processing Systems. pp. . https://proceedings.neurips.cc/paper_files/paper/1994/file/b8c37e33defde51cf91e1e03e51657da-Paper.pdf.

[15] Jean-Baptiste Mouret and Jeff Clune (2015). Illuminating Search Spaces by Mapping Elites. In arXiv preprint arXiv:1504.04909. https://arxiv.org/abs/1504.04909.

[16] Labella et al. (2006). Division of labor in a group of robots inspired by ants' foraging behavior. ACM Transactions on Autonomous and Adaptive Systems (TAAS). 1(1). pp. 4–25.

[17] Nitschke et al. (2008). Emergent specialization in biologically inspired collective behavior systems.

[18] Yao Qiang et al. (2024). Prompt Perturbation Consistency Learning for Robust Language Models. In Findings of the Association for Computational Linguistics: EACL 2024. pp. 1357–1370. https://aclanthology.org/2024.findings-eacl.91.

[19] Kellert, Stephen R (1996). The value of life: biological diversity and human society..

[20] Cadotte et al. (2011). Beyond species: functional diversity and the maintenance of ecological processes and services. Journal of applied ecology. 48(5). pp. 1079–1087.

[21] Shannon, Claude Elwood (1948). A mathematical theory of communication. The Bell system technical journal. 27(3). pp. 379–423.

[22] Dan Hendrycks et al. (2021). Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations. https://arxiv.org/abs/2009.03300.

[23] Stephanie Lin et al. (2022). TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 3214–3252. https://aclanthology.org/2022.acl-long.229.

[24] Maarten Sap et al. (2019). Social IQA: Commonsense Reasoning about Social Interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. pp. 4463–4473. https://aclanthology.org/D19-1454.

[25] Yonatan Bisk et al. (2020). PIQA: Reasoning about Physical Commonsense in Natural Language. In Proceedings of the AAAI Conference on Artificial Intelligence. pp. 7432–7439. https://arxiv.org/abs/1911.11641.

[26] Ankit Pal et al. (2022). MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical Domain Question Answering. In Proceedings of the Conference on Health, Inference, and Learning. pp. 248–260. https://arxiv.org/abs/2203.14371.

[27] Aida Amini et al. (2019). MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Human Language Technologies, Volume 1). pp. 2357–2367. doi:10.18653/v1/N19-1245. https://aclanthology.org/N19-1245.

[28] Jian Liu et al. (2020). LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence. pp. 3622–3628. https://arxiv.org/abs/2007.08124.

[29] Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168.

[30] David Rein et al. (2024). GPQA: A Graduate-Level Google-Proof Q&A Benchmark. In First Conference on Language Modeling. https://arxiv.org/abs/2311.12022.

[31] Shen-yun Miao et al. (2020). A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 975–984. https://aclanthology.org/2020.acl-main.92.

[32] Rowan Zellers et al. (2019). HellaSwag: Can a Machine Really Finish Your Sentence?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 4791–4800. https://aclanthology.org/P19-1472.

[33] Keisuke Sakaguchi et al. (2021). WinoGrande: An Adversarial Winograd Schema Challenge at Scale. Communications of the ACM. 64(9). pp. 99–106. https://arxiv.org/abs/1907.10641.

[34] Peter Clark et al. (2018). Think You Have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv preprint arXiv:1803.05457. https://arxiv.org/abs/1803.05457.

[35] Lianmin Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems. https://arxiv.org/abs/2306.05685.

[36] Austin et al. (2021). Program Synthesis with Large Language Models. arXiv preprint arXiv:2108.07732.

[37] Pichai et al. (2025). A new era of intelligence with gemini 3. Mountain View, CA: Google.

[38] Lee et al. (2025). Gemini embedding: Generalizable embeddings from gemini. arXiv preprint arXiv:2503.07891.