A Smaller Transformer in Your Transformer

Abstract

Recent findings indicate that Vision Transformers settle into locally similar computational phases, implying a level of depthwise computational redundancy. However, existing methods to exploit this redundancy either fail to reduce inference compute or severely degrade model expressivity. In this work, we formalise a unified view of block redundancy that decouples the geometry from specific surrogate interventions. We then introduce Transformer-Within-Transformer (TWT), a post-hoc method that fuses contiguous groups of redundant layers into a single learned surrogate layer. TWT reduces parameter count and inference compute while remaining competitive with original models using half the depth on natural images, and in several downstream histopathology settings, TWT matches or even improves on the original baseline.

Code available at: https://github.com/dsb-ifi/TWT.

Executive Summary: Vision Transformers have become a foundational architecture across modern artificial intelligence, yet their immense computational demands remain a major bottleneck for practical deployment. Recent studies show that contiguous sequences of layers within these models perform highly similar operations, revealing substantial depthwise redundancy. Prior attempts to exploit this redundancy have faced a difficult trade-off: recurrent approaches reduce model size by sharing parameters across layers but fail to reduce the actual computation required during inference, while linear approximations lower computational costs at the expense of severe drops in model expressivity.

The article sets out to evaluate whether these redundant sequences of layers can be compressed directly into single, non-recurrent surrogate layers without sacrificing dynamic token interactions or predictive performance.

To achieve this, the article introduces a post-hoc framework called Transformer-Within-Transformer. The method identifies contiguous redundant blocks of layers using dynamic programming based on representation similarity, selects the best-positioned initial candidate layer from each block via an auditioning procedure, and distils the original teacher network into the compressed student network in a single end-to-end training stage. The approach was systematically evaluated across natural image benchmarks, such as ImageNet-1k classification and ADE20k semantic segmentation, as well as digital histopathology tasks, including prostate cancer grading and breast cancer metastasis detection on whole-slide gigapixel images.

The analysis yielded several key findings. First, the proposed method reduces model depth and computational cost by approximately 50% while cutting parameter counts from roughly 85.7 million to between 29.0 and 36.1 million. Second, in natural image classification, the compressed models matched the accuracy of recurrent baselines while requiring only half the computational floating-point operations. Third, in histopathology applications, the compressed models not only retained baseline performance but frequently improved downstream accuracy and robustness, showing a 37.8% to 49.6% reduction in feature variance under input augmentations. Finally, for dense spatial prediction tasks like semantic segmentation, compressing layers caused a modest performance drop, demonstrating that fine-grained pixel-level tasks are more sensitive to layer removal than instance-level classification.

These findings indicate that the multi-step iterations in redundant transformer layers are largely an artifact of training dynamics rather than a functional necessity. In domain-shifted applications like histopathology, intermediate layers accumulate task-irrelevant noise; eliminating them acts as an effective regulariser that improves generalization. For organizations deploying large vision models, this compression offers significant cost reductions in computational hardware and energy without requiring full model retraining from scratch.

Organizations seeking to optimize large vision models should consider post-hoc block collapse as a practical alternative to uniform layer dropping or parameter-tied recurrence. Where immediate deployment on classification tasks is needed, models can safely be compressed to roughly half their original depth. However, before applying this method to dense spatial tasks such as segmentation, practitioners should run pilot evaluations to balance computational savings against acceptable loss in fine-grained accuracy.

The findings are supported by consistent results across multiple architectures and datasets, giving high confidence in classification settings. Nevertheless, readers should note that performance improvements over the uncompressed baseline were observed exclusively in domain-shifted histopathology contexts and should not be expected on standard natural image benchmarks. Further research is recommended to optimize block partitioning algorithms and explore pre-training methods that prevent redundant depth from developing in the first place.

1. Introduction

Section Summary: Vision Transformers process images through a uniform sequence of layers that often develop stretches of high similarity, creating computational redundancy where multiple layers perform overlapping work. Prior approaches have tried to exploit this either by reusing the same layer repeatedly to cut parameter counts or by replacing groups of layers with simpler approximations to reduce computation, but each comes with notable drawbacks in efficiency or accuracy. This work introduces a unified perspective on such redundancy and a post-training fusion technique called Transformer-Within-Transformer that merges similar blocks into single surrogate layers, trimming both parameters and inference cost while preserving or even improving performance on tasks like histopathology modeling.

Vision Transformers (ViTs) ([1, 2, 3]) have a particular architectural austerity; after the initial token embedding, each layer acts on the same token space, updating the representation through residual attention and feed-forward networks. Beyond its elegance ([4]), this architectural homogeneity exposes a central mode of investigation: how do layers organise to perform distinct computational tasks?

Recent discoveries ([5, 6]) show that ViTs tend to settle into depthwise-contiguous computational phases with a high degree of inter-layer similarity, implying a form of computational redundancy. While concurrent research agrees on the identification of the symptom, the methods used to exploit local depthwise redundancy vary from local recurrence ([6]), which lowers parameter counts but maintains the overall compute, to non-mixing approximations ([5, 7])—which reduce compute but incur a more prominent drop in performance.

Our work focuses on providing a more unified view of the phenomenon of block redundancy in ViTs and proposes a method that better exploits redundancy with minimal performance loss. Our contribution is threefold.

(i) We provide a formalisation of block redundancy in ViTs where recurrence and linear approximation appear as two sides of the same coin: both approximate local phases of computation whose constituent layers exhibit high functional similarity. (ii) We apply this paradigm with Transformer-Within-Transformer (TWT), a post-hoc method that fuses redundant blocks into a single surrogate layer, preserving the core ViT architecture. (iii) We show that TWT yields parameter reductions comparable to recurrent surrogates [6] while reducing executed inference computation, without incurring the performance degradation of simple linear surrogates. Notably, in histopathology foundation models, TWT improves downstream performance relative to the original model, and can even outperform larger baselines.

1.1 Note on Terminology and Nomenclature

The term block is used somewhat indiscriminately, denoting both the elementary computational unit of a ViT and, in discussions of depthwise redundancy, a contiguous group of such units. This quickly devolves into confusing references to blocks of blocks. To avoid ontological gymnastics, we elect to call the elementary ViT unit a layer: a multi-head attention operator and a feed-forward network. We reserve block to mean a contiguous sequence of layers, usually grouped by a homogeneity criterion, such as cosine similarity.

2. Block Redundancy in Vision Transformers

Section Summary: Vision transformers process inputs through theoretical computational phases, but in practice they learn these via empirical blocks of consecutive layers that often produce highly similar outputs. This block redundancy arises as an artifact of training rather than a fundamental requirement, allowing entire groups of layers to be approximated by simpler surrogate mappings without much loss in function. The specific choice of surrogate distinguishes different methods for exploiting this structure and forms the basis for more efficient model designs.

To formalise the procedural representational flow in ViTs, we distinguish between theoretical phases of computation and the empirical blocks of layers learned by a network. A phase is an ideal, theoretical stage of computation that executes a specific set of operations on the input. In practice, deep neural networks learn an approximation of these phases by distributing the computation across several learned layers. Evidence of these underlying phases emerges as block patterns in the network—contiguous groups of layers whose outputs exhibit high functional similarity. Previous work ([8, 5]) investigates the representational flow through these blocks, observing that updates tend to periodically decelerate, and some claim this implies an underlying recurrent program ([6]). Our objective is to understand this representational flow by characterising and exploiting inherent redundancies in empirical block structures that capture distinct computational phases of existing models.

We posit that the contiguous blocks observed in ViTs are geometrically redundant pathways navigating a noisy manifold, and their iterative behaviour is an artefact of the optimisation landscape rather than a strict computational necessity (cf. Figure 1). Consequently, these multi-step iterative blocks can be approximated by surrogate models that reflect these ideal phases. Our claim is that the particular choice of surrogate layer is the distinguishing factor between different approaches, and is the target of our formalisation of block redundancy.

ViT operators.

At its core, a transformer $F_\Theta$ can be decomposed as a set of transformations

$ F_\Theta = f_{\theta_L} \circ \cdots \circ f_{\theta_1}, $

where $f_{\theta_\ell} = f({\mkern 2mu\bm{\cdot}\mkern 2mu}; \theta_\ell)$ denotes a unique parametrisation for each layer $\ell$, and the full model parametrisation is given by $F_\Theta = F({\mkern 2mu\bm{\cdot}\mkern 2mu}; \Theta)$ with $\Theta = (\theta_1, \dots, \theta_L)$. Alternatively, Equation 1 can instead be formulated as contiguous block segments

$ F_\Theta = F_{\Theta_m} \circ \cdots \circ F_{\Theta_1}, $

where each subnetwork $F_{\Theta_j}$ for $1 \leq j \leq m$ contains layers with similar internal dynamics, i.e., some form of block redundancy. We begin by separating the phenomenon from the methods used to exploit it.

Block redundancy.

Let $B_j = [s_j, e_j]$ be a contiguous block of length $k_j = e_j - s_j + 1$, where $1 \leq s_j < e_j \leq L$. We say that block $B_j$ is $\varepsilon$-functionally-redundant if the layers within it implement highly similar transformations on the states visited by the model, such that

$ \max_{a, b \in B_j} \operatorname*{\mathbb{E}}{x \sim \mathrm{D}} \Big[ d\Big( f{\theta_a}\big(h_{a-1}(x)\big), f_{\theta_b}\big(h_{b-1}(x)\big) \Big) \Big] \le \varepsilon, $

where $h_{\ell}(x) = (f_{\theta_\ell} \circ \dots \circ f_{\theta_1})(x)$ denotes the intermediate hidden state after layer $\ell$, $d$ is a representation-space discrepancy measure, and $\mathrm{D}$ is a data distribution. In simple terms, a phase is block-redundant when all layers behave similarly across depth. Notably, [6] provide an analysis under $d(x,y) = 1-\cos(x,y)$. Empirical observations of the layer-to-layer distance similarity matrix

$ S_{a,b} = \operatorname*{\mathbb{E}}{x \sim \mathrm{D}} \left[ d\Big( f{\theta_a}\big(h_{a-1}(x)\big), f_{\theta_b}\big(h_{b-1}(x)\big) \Big) \right] $

reveal distinct block-diagonal structures. These structures define blocks $B_j$ which approximate the ideal phases of computation, where the intra-block representational similarity remains exceptionally high, as shown in Figure 2. This implies that $F_\Theta$ slowly integrates information over time to reach the final representations.

From redundancy to surrogate models.

Once a block has been identified as functionally redundant, we can form a stronger and more constructive query. Can an entire block be approximated by a simpler surrogate? Let

$ F_{\Theta_j} = f_{\theta_{e_j}} \circ \cdots \circ f_{\theta_{s_j}} $

denote the subnetwork implemented in block $B_j$ with parameters $\Theta_j = (\theta_{s_j}, \dots \theta_{e_j})$. We then seek a surrogate mapping $g_j$ such that

$ \operatorname*{\mathbb{E}}{x \sim \mathrm{D}} \Big[ d\Big( F{\Theta_j}\big(h_{s_j-1}(x)\big), g_{j}\big(h_{s_j-1}(x)\big) \Big) \Big] \le \varepsilon. $

The key distinction from previous approaches ([6, 5]) and our proposed TWT lies not in whether such a surrogate exists, but rather on the underlying assumptions motivating the choice of $g_j$.

2.1 Recurrent and Non-Mixing Surrogates

The Block Recurrent Hypothesis.

[6] operationalises block redundancy through a specific structural choice for $g_j$, and asks whether the surrogate can be chosen to be a repeated application of a single parameter-tied operator. Under this view, one writes

$ g_j = f_{\theta_j^*}^{k_j} = \underbrace{f_{\theta_j^*} \circ \cdots \circ f_{\theta_j^*}}_{k_j \text{ times}} = f^{k_j}({\mkern 2mu\bm{\cdot}\mkern 2mu}; \theta^*_j), $

such that the same parametrisation $\theta_j^*$ is reused $k_j$ times inside block $j$. Their proposed Raptor method optimises the parametrisation $\theta^*_j$ via

$ \theta^*j = \operatorname*{arg,min}\phi , \operatorname*{\mathbb{E}}{x \sim \mathrm{D}}\big\Vert F{\Theta_j}\big(h_{s_j-1}(x)\big) - f^{k_j}\phi\big(h{s_j-1}(x)\big) \big\Vert_2^2, $

which fine-tunes an existing model via a recurrent objective, noting that Raptor performs this optimisation in two separate training stages.

While Raptor demonstrates that contiguous phases can be approximated by shared-weight layers, we argue that recurrence is an overly specific interpretation of a broader geometric phenomenon. In a residual architecture, small functional updates along a locally flat computational phase naturally induce block-diagonal similarity. A sequence of independent layers traversing such a phase can therefore yield a similar signature as a recurrent operator, without implementing repeated dynamics or requiring identical weights. From this perspective, recurrence is a useful parameter-sharing regulariser imposed on redundant phases, but not the fundamental mechanism generating them. And because it still executes all $k_j$ transformations, computations at inference remain largely intact.

Non-mixing Surrogates.

TOAST ([5]) takes the opposite approach to BRH@. Rather than imposing recurrence, it bypasses redundant phases entirely by fitting a closed-form linear map between their endpoints. Similarly, NOSE ([7]) replaces specific ViT layers exclusively with feed-forward networks, adding more expressivity at the cost of more compute. While both reduce parameters and raw compute without retraining, this strategy also limits the expressivity of the surrogate $g_j$.

A key property of attention operators is their capability for dynamic token-mixing, which provides a high degree of expressivity ([9, 10]). A linear map can somewhat preserve proximity, but it cannot meaningfully reproduce token interactions between layers in identified blocks. While TOAST and NOSE demonstrate that entire phases can be fused using strictly less-expressive operators, they produce models that cannot retain the expressivity of their original components.

2.2 Transformer-Within-Transformer

TWT looks to resolve the tension between surrogate expressivity and inference efficiency. We hypothesise that if a contiguous block $B_j = [s_j, e_j]$ approximates a single ideal computational phase, the intermediate representations, $h_{s_j}(x), \dots, h_{e_j-1}(x)$, are not strict necessities, but transitional micro-steps across a locally flat manifold. Therefore, the surrogate $g_j$ can be chosen as a single, non-recurrent operator $f_{\phi_j}$ that computes the phase directly via

$ h_{e_j}(x) \approx f_{\phi_j}\big(h_{s_j-1}(x)\big). $

Crucially, to retain dynamic token-mixing, we restrict $f_{\phi_j}$ to the standard architecture of a ViT layer: a multi-head attention mechanism and a feed-forward network. In replacing a redundant block with exactly one layer, TWT fuses the computational phase into a single step, circumventing the iterative inference cost retained by BRH ([6]).

This formalisation fundamentally alters the algorithmic interpretation of ViT depth. It implies that deep networks do not strictly require $k_j$ distinct attention passes to incrementally route information within a phase. Instead, a single, optimally parameterised attention and feed-forward pass possesses sufficient representational capacity to execute the entire spatial and channel-wise reorientation required for that ideal phase.

3. Transformer-Within-Transformer Surrogate Discovery

Section Summary: The section introduces a framework for compressing Vision Transformers by identifying groups of redundant layers that perform similar computations and replacing each group with a single learned surrogate layer. This is done through three steps: using dynamic programming to find suitable layer blocks based on a discrepancy threshold, selecting a strong initial candidate for each surrogate by testing existing layers or their average, and then training the resulting compact model end-to-end with progressively introduced supervision signals at multiple depths. The approach avoids the repeated layer applications or restrictive linear approximations used in prior methods, enabling faster inference while preserving accuracy.

To operationalise block redundancy, we propose a pruning and distillation framework that condenses contiguous blocks of learned operators into a single algorithmic step reflecting the ideal phase. Unlike the BRH, which requires a surrogate operator $g_j = f_{\theta_j^*}^{k_j}$ to iteratively unroll $k_j$ times to mimic a target block $B_j$, our method approximates a block's computational phase with a single application of $f_{\phi_j}$ without recurrent iteration, as formalised in Equation 2. We achieve this through a three-step process: (1) dynamic block discovery, (2) optimal candidate initialisation, and (3) distillation via single-stage deep supervision. We depict this process and contrast it against the literature in Figure 3.

3.1 Block Discovery via Max-Min Dynamic Programming

Our first objective is to identify contiguous subsets of layers that form $\varepsilon$-functionally-redundant blocks. Let $F_\Theta$ be a trained Vision Transformer with $L$ layers. We compute the discrepancy matrix $S \in \mathbb{R}^{L \times L}$, where $S_{a,b}$ is the expected discrepancy $d$ of the token representations $h_a(x)$ and $h_b(x)$ over a representative calibration set $\mathrm{D}$, as introduced in Section 2. In practice, we let $d$ be the cosine distance between intermediate activations.

We frame block discovery as a partitioning problem over depth. We seek a set of contiguous blocks $\mathrm{P} = {B_1, \dots, B_m}$ that cover the entire network, where each $B_j = [s_j, e_j]$ and $s_{j+1} = e_j + 1$. A valid partition must satisfy a maximum intra-block discrepancy threshold $\varepsilon$, ensuring that the boundary discrepancy satisfies $S_{s_j, e_j} \le \varepsilon$ for all $j$.

To prevent catastrophic degradation, we employ a min-max dynamic programming approach. We first minimise the total number of blocks $m$, effectively maximising compression. To break ties among equally minimal partitions, we select the partition that minimises the worst-case discrepancy among its blocks. Formally, we optimise

$ \min_{\mathrm{P}} \left( m, \max_{j \in {1 \dots m}} S_{s_j, e_j} \right) \quad \text{s.t.} \quad S_{s_j, e_j} \le \varepsilon, \forall j. $

This yields a deterministic merge plan where each block $B_j$ corresponds to a single ideal phase to be captured by a surrogate operator. We provide a sensitivity analysis in Appendix D showing how varying the threshold $\varepsilon$ controls the granularity of this partition.

3.2 Macro-Step Initialisation via Auditioning

Instead of initialising a surrogate operator $f_{\phi_j}$ randomly, we exploit the parameters the teacher has already learned. For a given block $B_j = [s_j, e_j]$, we generate a pool of candidate operators. This pool, denoted as $\mathrm{C}j = { f{\theta_\ell}: \ell \in B_j } \cup { \bar{f}_j }$, includes each individual layer from the teacher's block, as well as an averaged operator $\bar{f}_j$ constructed by averaging the weight matrices of all layers in the span (while keeping normalisation parameters isolated). We evaluate each candidate on a small calibration batch to minimise the local mapping error

$ f^*_{\phi_j} = \operatorname*{arg,min}{f \in \mathrm{C}j} \operatorname*{\mathbb{E}}{x \sim \mathrm{D}} \left[ \left\Vert f(h{s_j-1}(x)) - h_{e_j}(x) \right\Vert_2^2 \right]. $

This auditioning process identifies the operator best naturally positioned to execute the macro-step, significantly stabilising early training dynamics.

We provide an ablation study in Appendix C showing that the auditioned candidate generally outperforms the worst candidate, supporting the value of auditioning for downstream transfer.

3.3 Distillation via Deep Supervision

With the collapsed architecture initialised, we distil the teacher into the student without recurrent rollouts or linear surrogates. Instead of the dual-stage training procedure used in Raptor ([6]), we stitch the $K$ surrogate blocks together into a unified student model $F_{\Phi}$ and optimise it end-to-end.

Let $\mathrm{B}={(s_j,e_j)}{j=1}^{K}$ denote the merge plan, where student block $j$ replaces teacher blocks $s_j,\dots,e_j$. For an input image $x$, let $h_j(x;\Phi)$ be the output of student block $j$, and let $h{e_j}(x;\Theta)$ be the output of the last teacher block in the corresponding teacher segment. Let $F_{\Phi}(x)$ and $F_{\Theta}(x)$ denote the final pre-classification backbone features of the student and teacher, respectively. To gradually introduce deeper supervision terms, we use a staggered cosine schedule over normalised training time $\tau \in [0,1]$:

$ \begin{aligned} \omega_j(\tau)= \begin{cases} \frac{1-\cos!\left(\pi \tau / \alpha_j\right)}{2}, & 0 \le \tau < \alpha_j,\ 1, & \alpha_j \le \tau, \end{cases} \end{aligned} $

where $\alpha_j \in (0,1]$ is the activation time assigned to block $j$, with shallower blocks activated earlier and deeper blocks later. The training objective is

$ \mathrm{L}(x,\tau)

\sum_{j=1}^{K} \omega_j(\tau), \mathrm{MSE}!\left(h_j(x;\Phi),, h_{e_j}(x;\Theta)\right) + \omega_{K+1}(\tau), \mathrm{MSE}!\left(F_{\Phi}(x),, F_{\Theta}(x)\right). $

Thus, the student is supervised both at intermediate block outputs and at the final representation, with deeper losses introduced progressively over training. All student parameters are optimised jointly using AdamW@.

Layer Scaling.

While most pre-trained models use LayerScale ([11]) during initial optimisation, these parameters are not always included in pre-trained checkpoints. Our experiments indicate that several pre-trained models have a marked increase in norms for later layers, particularly in natural image models, as seen in Figure 4. To improve convergence, we explicitly introduce LayerScale, initialised as identity to allow the model to more easily adapt norms to dropped layers in block fusion. We find that a single scalar parameter typically suffices to improve convergence during fitting. These are fused with existing parameters in the final model weights.

4. Geometric Evidence of Block Collapse

Section Summary: The section examines whether replacing redundant blocks with single surrogate layers preserves a network’s overall representational flow by tracking the intrinsic dimension of token representations across depth. In both histopathology and natural-image models, the pruned student follows the teacher’s coarse trajectory at block boundaries yet shows markedly lower intrinsic dimension and substantially reduced variance under input augmentations. These geometric patterns indicate that the removed intermediate layers mainly accumulate noise rather than introduce distinct computational stages, yielding a more stable and compact representation.

The formulation in Section 2.2 posits that a single learned surrogate layer can approximate a contiguous redundant block. To test this hypothesis, we examine how collapse alters the representation space. If intermediate layers within a block primarily refine a shared computational phase rather than introducing a new representational stage, the pruned student network should preserve the teacher's coarse trajectory at block boundaries. We quantify this geometric alignment by evaluating the intrinsic dimension (ID) of the token representations across depth using the Two-NN estimator [12].

We compare an unpruned teacher model using the H0-mini [13] backbone for histopathology images and DINOv2 [2] for natural images against the corresponding pruned student networks. To capture the full scope of the representational flow, we track the ID of the global class token, the spatially averaged patch tokens, and the raw, unpooled flattened patch token sequence.

4.1 Intrinsic Dimension and Smoothing

The raw, flattened token representation reveals geometric changes that are less apparent after pooling. Across block boundaries, the student follows the same broad ID trajectory as the unpruned teacher, indicating that the surrogate layers preserve the coarse representational progression of the original network. At the same time, the student consistently has a lower ID in the histopathology setting, particularly for the flattened patch-token representation in the rightmost panel of Figure 5. This pattern suggests that block collapse removes part of the high-dimensional variation introduced by the teacher's intermediate layers while retaining the endpoint geometry.

4.2 Variance Reduction Under Stochastic Augmentation

We next evaluate whether the lower ID observed in the student is accompanied by greater stability under input perturbations. For each input, we apply stochastic spatial and colour augmentations and measure the standard deviation of the resulting token representations at block boundaries. The shaded regions in Figure 5 represent the variance across these augmentation passes, showing larger variation for the unpruned teacher at several intermediate boundaries. We quantify this instability in Table 1.

::: {caption="Table 1: Intrinsic Dimension (ID) and Standard Deviation (STD) of Flattened Tokens at Block Boundaries (Overall Combined: Clean + Augmented). The unpruned teacher is H0-mini, while the block-collapsed student is the pruned H0-mini obtained by replacing the discovered blocks with 6 surrogate layers. The student network shows consistently lower ID and reduced variance, indicating lower sensitivity to the sampled augmentations."}

:::

As shown in Table 1, the block-collapsed student demonstrates a substantial reduction in representational standard deviation—ranging from 37.8% to 49.6% across all boundaries compared to the teacher network. This indicates that the collapsed student is less sensitive to augmentations at each measured boundary. Since the student still follows the teacher's coarse ID trajectory, the lower variance suggests that the surrogate layers preserve the phase boundaries while reducing augmentation-sensitive variation along the intermediate path.

This can be interpreted as evidence for the claim in Section 3. Intermediate layers within a redundant block are not contributing distinct computational stages, but instead accumulate stochastic variation as they traverse a locally flat region of the representation manifold. By removing these transitional steps, TWT can provide a more compact, stable representation with less sensitivity to perturbations by augmentation. The downstream results in Table 2 and Table 3 confirm that this source of variation carries little-to-no task-relevant signal and is likely noise introduced by over-provisioned depth.

5. Experiments

Section Summary: The authors evaluate their TWT compression method on both natural images and large histopathology whole-slide images, testing whether collapsing redundant blocks in vision backbones can cut parameters and computation with minimal accuracy loss. They apply it to two domain-specific models and compare against other pruning techniques, training slide-level classifiers on prostate and breast cancer datasets to measure how well the reduced models perform on internal and external test sets. Results show TWT often matches or exceeds baseline performance while using roughly one-third to one-half the active parameters, with similar methodology applied to natural-image models.

We evaluate TWT across two distinct visual domains: natural images and histopathology Whole-Slide Image (WSI) classification. Our primary objective is to demonstrate that collapsing redundant blocks not only reduces parameter counts and computation but also does so with little to no performance loss. We compare TWT against Raptor ([6]) via BRH and two state-of-the-art depth pruning methods: WDPruning [14] and NOSE [7], where each method is applied to a baseline model (reported as such in each result for reference).

5.1 Histopathology Models

Histopathology models process gigapixel whole-slide images (WSIs), where inference cost scales with the number of retained tissue tiles per slide. We, therefore, evaluate whether TWT preserves slide-level performance after reducing the depth of two domain-specific DINOv2-style backbones: H0-mini [13] and Hibou-B [15].

We use PANDA training tiles for histopathology phase discovery and backbone compression. We then freeze each pruned or unpruned backbone and precompute tile embeddings for all downstream cohorts. Each WSI forms a bag of foreground tile embeddings, and we train slide-level MIL heads on these frozen bags. We evaluate two MIL aggregators, ABMIL [16] and TransMIL [17], to test whether the compressed features remain useful across different slide-level pooling mechanisms.

For prostate cancer grading, we train MIL heads on PANDA [18] using the split from Song et al. [19]. We select checkpoints by PANDA validation QWK and report QWK ($\times100$) on PANDA test and TCGA-PRAD [20]. For breast metastasis detection, we train MIL heads on CAMELYON17 [21] using corrected binary labels from Ling et al. [22]. We select checkpoints by the 5-epoch moving average of CAMELYON17 validation accuracy and report slide-level accuracy on CAMELYON17 test and CAMELYON16 [23]. We provide tiling, tissue filtering, label processing, and optimisation details in Appendix A.

::: {caption="Table 2: Histopathology MIL results for H0-mini. We report QWK (×100) for prostate grading and accuracy for breast metastasis detection. Prostate MIL heads train on PANDA and test on PANDA/TCGA-PRAD; metastasis MIL heads train on CAMELYON17 and test on CAMELYON17/CAMELYON16. Deltas compare against the uncompressed backbone with the same MIL head."}

:::

::: {caption="Table 3: Histopathology MIL results for Hibou-B. We report QWK (×100) for prostate grading and accuracy for breast metastasis detection. Prostate MIL heads train on PANDA and test on PANDA/TCGA-PRAD; metastasis MIL heads train on CAMELYON17 and test on CAMELYON17/CAMELYON16. Deltas compare against the uncompressed backbone with the same MIL head."}

:::

Pruning Implementation Details.

We train pruned backbones using AdamW [24], a cosine learning-rate decay [25], and gradient clipping at norm 1.0. We use a peak learning rate of $3\times10^{-4}$. We employ colour jitter and Gaussian blur data augmentations during distillation. Notably, we omit both the auxiliary LayerScale parameters and the staggered cosine loss schedule for these histopathology models. Unlike natural image backbones, they do not exhibit late-layer norm explosion and optimise stably without these additions. We scale training budgets by student size: for H0-mini, we train NOSE and WDPruning for 2 epochs, Raptor for 3 epochs total (1 in Stage-1, 2 in Stage-2), and TWT for 2 epochs (depth 4) or 1.5 epochs (depth 5). For Hibou-B, we use the same hyperparameters as for H0-mini across all methods, with 2 epochs for TWT at both depths. We also experimented with extended training budgets on Hibou-B (up to 5 epochs for NOSE and WDPruning, and 3 epochs for Raptor); however, these extended runs yielded similar downstream outcomes, confirming that the performance bottlenecks are not due to insufficient training time.

H0-mini results.

Table 2 shows that TWT preserves prostate grading performance while reducing the active backbone size by more than half. The five-layer TWT model uses $36.11$M parameters instead of $85.74$M and matches the uncompressed model on PANDA for both ABMIL and TransMIL. It also improves TCGA-PRAD QWK for both MIL heads, with gains of $+0.94$ and $+5.87$, respectively. The four-layer model further reduces the active parameter count to $29.02$M and remains close to the baseline on PANDA while still improving TCGA-PRAD. On CAMELYON17, TWT improves accuracy across both MIL heads and both depths. CAMELYON16 shows a small drop relative to the baseline, but the pruned models remain within roughly two percentage points while using about one third of the active backbone parameters.

Hibou-B results.

Table 3 evaluates whether the same compression behaviour transfers to a second histopathology foundation model. TWT again provides a strong parameter–performance trade-off: the four-layer model uses $29.02$M active parameters compared with $85.74$M for the full backbone, while remaining close to the baseline on PANDA and improving ABMIL performance on TCGA-PRAD. On CAMELYON, TWT is particularly strong for the external CAMELYON16 evaluation, indicating that the collapsed backbone can retain transferable features across tissue type and dataset shift. Some competing pruned baselines, such as WDPruning and NOSE, are less stable in this setting, especially at aggressive compression levels, but our TWT achieves similar parameter-performance trade-offs while fundamentally reducing inference compute.

5.2 Natural Image Models

Our experiments with natural images closely follow the pruning methodology from Section 5.1, with few exceptions. For natural images, we use a total of 8 epochs across all methods, and while training stabilises quite early, we see a more dramatic effect from using a staggered cosine schedule for the loss function than in histopathology. Our experiments focus on ImageNet-1k ([26]) for classification and ADE20k ([27]) for segmentation. Evaluation protocols follow the respective baselines ([28, 29, 3]).

::: {caption="Table 4: Natural image classification results for base-capacity backbones. We report top-1 accuracy at 224 × 224. Deltas compare against the uncompressed backbone of the same model."}

:::

Classification.

Table 4 shows that TWT achieves competitive accuracy at a fraction of the inference cost. At six layers, TWT matches or closely tracks Raptor across all three backbones while operating at roughly half the FLOPs; on DINOv2 B/14, the six-layer TWT model outperforms Raptor on all folds despite requiring only 11.71 GFLOPs versus 23.42 GFLOPs. Interestingly, Raptor ([6]) provides marginal accuracy improvements on B/16 models, notably with no reduction in compute. With fewer tokens, the attention matrix is smaller ($196\times 196$ vs. $256\times 256$), so each recurrent application achieves more complete global mixing relative to the total information content. A recurrent layer could have a smoother optimisation target simply because the token-mixing space has a lower dimensionality.

Compared to WDPruning, which operates at a near-identical compute budget, TWT consistently matches or improves accuracy, suggesting that the phase-aware surrogate initialisation and deep supervision yield a more effective model than uniform depth pruning. At five layers, all methods degrade more noticeably, reflecting compression that likely goes beyond the redundant phases into functionally distinct computation. Nevertheless, TWT remains within approximately one point of the baseline across all three tested models.

In contrast to the histopathology results (Table 2 and Table 3), no method improves over the uncompressed baseline on natural images. This is consistent with our observations on smoothing from Section 4.1; ImageNet backbones were trained on this distribution, so the redundant phases are already well-tuned and collapsing them can at best preserve performance. In histopathology, the backbone operates out-of-distribution relative to its pre-training data, and the intermediate iterative steps accumulate domain-irrelevant noise that block collapse actively removes.

Dense Tasks.

In addition to instance-level classification, we evaluate semantic segmentation on ADE20k, comparing against baseline DINO-family models and Raptor ([6]). Table 5 shows that dense prediction is sensitive to block approximation even when the full recurrent execution cost is retained. Raptor preserves the original network's iterative computation but still incurs a modest drop in mIoU. In comparison, TWT reduces inference compute by approximately $50%$, at the cost of an additional 1.3 and 1.7 mIoU on DINOv2 and DINOv3, respectively.

This result is consistent with the distinction made by [6] between patch-token and instance-level dynamics. Dense prediction depends directly on the spatial token field, whereas classification can remain robust when the instance representation is preserved. A recurrent surrogate may therefore retain local patch-token evolution more faithfully because it maintains iterative token mixing. The additional mIoU loss therefore reflects a trade-off between preserving patch-level trajectory structure and eliminating recurrent inference cost, suggesting that dense prediction may require less aggressive block collapse. We discuss this trade-off further in Section 7.1.

::: {caption="Table 5: Semantic segmentation results on ADE20k. Deltas compare against the uncompressed baseline."}

:::

6. Related Work

Section Summary: Existing research models network depth as continuous dynamical systems that naturally form attractors, leading to methods that apply recurrent tied layers, prune ineffective blocks, or distill internal trajectories to reduce redundancy. These approaches often treat multi-step processing as essential or remove layers piecemeal based on heuristics. By comparison, the TWT perspective frames such repetition as an artifact of training and replaces contiguous blocks with a single fused computation step through a technique called block collapse.

Depth as Recurrent Dynamics.

Existing work conceptualises deep networks as continuous dynamical systems where representations naturally cluster into meta-stable attractors as they propagate through depth [30, 31, 32]. BRH exploits this simplicity bias by approximating contiguous blocks with a single parameter-tied layer applied recurrently [6], operating on the assumption that multi-step iterative execution is strictly necessary. In contrast, we argue that this multi-step behaviour is merely an artefact of the optimisation landscape. Instead of unrolling a recurrent operator, TWT suggests that these redundant blocks approximate a single, ideal phase of computation that maps inputs directly to the terminal representation in a fused step.

Layer Relevance and Structural Pruning.

To handle block redundancy, current methods often identify "ineffective" layers and excise them using structural pruning [33, 34, 35], accuracy-grounded relevance metrics [36], or structural linearisation of consecutive blocks [37, 38, 39] to salvage downstream performance. Rather than simply dropping layers based on heuristics or generic proxies, we frame this multi-step processing explicitly as executable redundancy through the lens of TWT. This theoretically justifies replacing an entire contiguous sequence of layers with a single step instead of selectively pruning isolated parts.

Trajectory Distillation and Block Collapse.

Advanced model compression techniques move beyond simple logit matching by distilling internal hidden-state trajectories (using metrics like CKA or value-relations) [40, 41] or by iteratively substituting full layers with retrained compact blocks [42]. Building upon this, we introduce "block collapse" as a localised distillation mechanism guided by TWT. By anchoring the start and end of a phase through deep supervision, we replace the entire redundant block with a single, non-recurrent surrogate operator. This forces the network to bypass intermediate representational dawdling, actively denoising the manifold and directly yielding inference-time compute savings.

7. Conclusion

Section Summary: In this study, researchers examined unnecessary repetition in the layers of vision transformer models and found that many intermediate computations could be skipped without harming overall results. They developed a new approach called Transformer-Within-Transformer that identifies and replaces these repetitive phases with single, learned layers, cutting down on model size and processing time while keeping or even improving accuracy in medical image tasks. The work also notes that gains are not guaranteed in every setting and calls for further research into why such repetition occurs in the first place.

In this work, we investigated the phenomenon of block redundancy in Vision Transformers. While recent frameworks like BRH interpret highly similar contiguous layers as evidence of underlying recurrence, our analyses demonstrate that these iterative micro-steps are not strictly computationally necessary. Our intrinsic dimension and variance analyses further suggest that, in the histopathology setting, intermediate steps within redundant phases can accumulate augmentation-sensitive variation without altering the coarse representational trajectory.

To resolve this inefficiency, we introduced Transformer-Within-Transformer (TWT). By dynamically identifying redundant phases and collapsing them into single, learned surrogate layers, TWT replaces intermediate iterations with a single macro-step. Unlike linear approximations that sacrifice token-mixing expressivity, or block-recurrent methods that still execute costly iterations at inference, our approach tracks the coarse geometric trajectory while reducing both parameter count and inference compute. Across natural-image models, TWT preserves competitive downstream performance at substantially reduced compute, while in several histopathology settings it matches or improves over the uncompressed baseline.

7.1 Limitations and Further Work

While TWT provides a general methodology for exploiting block redundancy in ViTs, it is not to be interpreted as a universal panacea. As discussed in Table 2, Table 3, and Table 4, performance improvement over the uncompressed baseline appears only in specific settings, notably the histopathology models in our experiments. This does not imply that a performance increase should be expected in the general case. Dense prediction remains more sensitive to block collapse than instance-level classification. Our segmentation results suggest that patch-level tasks benefit from preserving more of the intermediate token trajectory, and may therefore require less aggressive collapse or task-aware distillation.

The partitioning scheme in Section 3.1 remains an approximation. Better block discovery improves performance, motivating future work on submodular optimisation or optimal transport for this combinatorial problem.

Finally, we focus on post-hoc exploitation of block redundancy. Explaining how such redundancy arises in vision models, and how it might be mitigated during pre-training, remain important directions for future work.

Acknowledgments

This work was funded by the Research Council of Norway through Visual Intelligence, Centre for Research-based Innovation (309439), and by the South-Eastern Norway Regional Health Authority (2024039). The computations were performed on resources provided by Sigma2 (NN8104K) — the National Infrastructure for High-Performance Computing and Data Storage in Norway. We acknowledge Sigma2 for access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium through Sigma2, Norway.

Appendix

Section Summary: The appendix outlines experimental protocols for training and evaluating multiple-instance learning models on prostate cancer grading and breast metastasis detection tasks, drawing on datasets such as PANDA, TCGA-PRAD, and CAMELYON with standardized slide tiling, label conversion, feature extraction from frozen backbones, and optimization using Adam and weighted cross-entropy. It further specifies a pruning workflow that identifies phase boundaries from PANDA tiles alone, then applies distillation to create compressed student models, along with parameter-count adjustments for inference. Supporting analyses examine the impact of surrogate-layer initialization choices and the sensitivity of the geometric discovery threshold on resulting model partitions and performance.

A. Histopathology Experimental Details

Datasets and splits.

For prostate cancer grading, we train MIL heads on PANDA [18] and use the train/validation/test split from Song et al. [19]. We use the validation split for checkpoint selection and report final performance on the PANDA test split. We use TCGA-PRAD [20] only as an external evaluation cohort. For breast metastasis detection, we train MIL heads on CAMELYON17 [21] and evaluate on the CAMELYON17 test split and CAMELYON16 [23]. We use corrected CAMELYON slide labels from Ling et al. [22].

Label processing.

PANDA provides slide-level ISUP grade labels. For TCGA-PRAD, we convert Gleason primary and secondary patterns to ISUP grade groups using the standard mapping: Gleason $3+3$ maps to grade group 1, $3+4$ to 2, $4+3$ to 3, total score 8 to 4, and total score 9–10 to 5. We assign benign or non-cancer cases to grade group 0 when applicable. CAMELYON17 and CAMELYON16 use binary slide-level labels for metastasis detection.

Tiling and tissue filtering.

We tile WSIs into non-overlapping $256\times256$ patches. For PANDA and TCGA-PRAD, we retain tiles with at least $60%$ foreground tissue. For CAMELYON17 and CAMELYON16, we identify tissue regions using Otsu thresholding [43]. During feature extraction, we resize tiles to the input resolution expected by each foundation model and apply the corresponding model-specific normalisation statistics.

Feature extraction.

We freeze each pruned or unpruned backbone before MIL training and precompute tile embeddings for every downstream cohort. For DINOv2-style backbones with register tokens, we concatenate the class token with the mean of the patch tokens and exclude the four register tokens. This produces a $1536$-dimensional tile embedding. Each WSI bag contains all retained foreground tile embeddings.

MIL models.

We train ABMIL [16] and TransMIL [17] on frozen tile embeddings with an identity encoder. For prostate grading, each MIL head predicts six ISUP classes. For metastasis detection, each MIL head predicts two classes. We use class-weighted cross-entropy for all MIL experiments.

Optimisation and checkpoint selection.

For PANDA, we train each MIL head for 20 epochs using Adam with learning rate $10^{-4}$, weight decay $10^{-4}$, cosine annealing, batch size of one WSI, gradient clipping at norm 1, and gradient accumulation over 32 steps. We select the checkpoint with the highest PANDA validation QWK and evaluate it on PANDA test and TCGA-PRAD.

For CAMELYON17, we train each MIL head for 100 epochs using Adam with learning rate $10^{-4}$, weight decay $10^{-4}$, cosine annealing, class-weighted cross-entropy, and gradient clipping at norm 1. We evaluate after every epoch on CAMELYON17 validation, CAMELYON17 test, and CAMELYON16. We select the checkpoint with the highest 5-epoch moving average of CAMELYON17 validation accuracy and report the corresponding CAMELYON17 and CAMELYON16 accuracies.

Metrics.

For PANDA and TCGA-PRAD, we report quadratic weighted kappa (QWK) multiplied by 100. For CAMELYON17 and CAMELYON16, we report slide-level accuracy in percentage points. We report the mean and standard deviation across independent MIL training runs.

B. Histopathology Pruning Protocol

Phase discovery and pruning data.

For the histopathology experiments, we compute phase boundaries using PANDA training tiles only. We sample $10{,}000$ tiles and compute layer-wise cosine distances between intermediate representations. We use the resulting phase boundaries to define the pruned TWT students. All pruned backbones, including TWT and the baselines, are trained by distillation on PANDA training tiles with the corresponding unpruned backbone frozen as the teacher. We do not use TCGA-PRAD, CAMELYON17, or CAMELYON16 for phase discovery or backbone pruning.

Optimisation.

We train pruned backbones with feature-level MSE distillation, AdamW, cosine learning-rate decay, and gradient clipping. Unless stated otherwise, we use a learning rate $3\times10^{-4}$. NOSE, WDPruning, and TWT use single-stage distillation. Raptor uses its standard two-stage protocol: recurrent block training followed by stitched model fine-tuning.

Parameter counts.

Parameter counts in the main tables refer to the feature extractor used at inference. For WDPruning, this includes only the retained first $K$ layers and the single depth-specific probe used for feature extraction. For NOSE, this uses the unbloated checkpoint after removing pruned attention parameters. For Raptor, this uses the recurrent feature extractor executed at inference.

C. Ablation on Surrogate Initialisation

In the main text, we describe an auditioning process to select the optimal layer from the teacher's block to initialise the student's surrogate layer. To isolate the effect of this initialisation, Table 6 ablates the procedure by comparing models initialised with the best candidate versus the worst candidate within the block. The results show that the auditioned candidate generally provides stronger downstream transfer than the worst candidate, although the effect is not uniform across every dataset and compression depth, supporting the value of the auditioning step.

::: {caption="Table 6: Comparison of TWT initialised with the best vs. worst candidate block across all evaluated depths. We report QWK (× 100) for prostate grading and accuracy for breast metastasis detection. Deltas compare against the baseline model."}

:::

D. Sensitivity of Discovery Threshold $\varepsilon$

The threshold $\varepsilon$ serves as the core hyperparameter for dynamic block discovery. Rather than a direct training parameter, it acts as a geometric distance bound: smaller $\varepsilon$ enforces strict boundary requirements, yielding highly granular partitions with many retained blocks, whereas larger values lead to more collapse. Figure 6 illustrates this monotonic relationship across our four evaluated backbones, showing how varying $\varepsilon$ affects the count of retained blocks.

References

Section Summary: The references section compiles a list of academic papers and preprints that underpin research on transformer models, attention mechanisms, and self-supervised learning techniques. These citations draw heavily from major machine learning conferences and journals, covering foundational work on vision transformers alongside more specialized studies in model efficiency, recurrence, and medical image analysis. The entries span from classic contributions in 2017 to recent advances in 2025 and 2026.

[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Adv. Neural Inf. Process. Sys. (NeurIPS), volume 30, pages 5998–6008, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.

[2] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: learning robust visual features without supervision. Trans. Mach. Learn. Res., 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=a68SUt6zFt. Featured Certification.

[3] Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. Dinov3, 2025. URL https://arxiv.org/abs/2508.10104.

[4] Avi Schwarzschild, Arjun Gupta, Amin Ghiasi, Micah Goldblum, and Tom Goldstein. The uncanny similarity of recurrence and depth. In Inter. Conf. Learn. Represent. (ICLR), 2022. URL https://openreview.net/forum?id=3wNcr5nq56.

[5] Irene Cannistraci, Simone Antonelli, Emanuele Palumbo, Thomas M. Sutter, Emanuele Rodolà, Bastian Rieck, and Julia E Vogt. TOAST: Transformer optimization using adaptive and simple transformations. Trans. Mach. Learn. Res., 2026. ISSN 2835-8856. URL https://openreview.net/forum?id=fSwMCsBtTG.

[6] Mozes Jacobs, Thomas Fel, Richard Hakim, Alessandra Brondetta, Demba E. Ba, and T. Anderson Keller. Block recurrent dynamics in vision transformers. In Inter. Conf. Learn. Represent. (ICLR), 2026. URL https://openreview.net/forum?id=gH3HhnfWLC.

[7] Sihao Lin, Pumeng Lyu, Dongrui Liu, Tao Tang, Xiaodan Liang, Andy Song, and Xiaojun Chang. MLP can be a good transformer learner. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), pages 19489–19498, 2024.

[8] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Inter. Conf. Mach. Learn. (ICML), 2019. URL https://proceedings.mlr.press/v97/kornblith19a.html.

[9] Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self-attention and convolutional layers. In Inter. Conf. Learn. Represent. (ICLR), 2020. URL https://openreview.net/forum?id=HJlnC1rKPB.

[10] Jorge Pérez, Pablo Barceló, and Javier Marinkovic. Attention is turing complete. J. Mach. Learn. Res., 2021.

[11] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Herve Jegou. Going deeper with Image Transformers . In IEEE Inter. Conf. Comput. Vis. (ICCV), pages 32–42, 2021. doi:10.1109/ICCV48922.2021.00010. URL https://doi.ieeecomputersociety.org/10.1109/ICCV48922.2021.00010.

[12] Elena Facco, Maria d'Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Sci. Rep., 7(1):12140, 2017.

[13] Alexandre Filiot, Nicolas Dop, Oussama Tchita, Auriane Riou, Rémy Dubois, Thomas Peeters, Daria Valter, Marin Scalbert, Charlie Saillard, Geneviève Robin, et al. Distilling foundation models for robust and efficient models in digital pathology. In IEEE Inter. Conf. Med. Image Comput. Comput. Assist. Interv. (MICCAI), pages 162–172. Springer, 2025.

[14] Fang Yu, Kun Huang, Meng Wang, Yuan Cheng, Wei Chu, and Li Cui. Width & depth pruning for vision transformers. In AAAI Conf. Artif. Intell. (AAAI), volume 36, pages 3143–3151, 2022.

[15] Dmitry Nechaev, Alexey Pchelnikov, and Ekaterina Ivanova. Hibou: A family of foundational vision transformers for pathology. arXiv preprint arXiv:2406.05074, 2024.

[16] Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In Inter. Conf. Mach. Learn. (ICML), pages 2127–2136. PMLR, 2018.

[17] Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, et al. TransMIL: Transformer based correlated multiple instance learning for whole slide image classification. In Adv. Neural Inf. Process. Sys. (NeurIPS), volume 34, pages 2136–2147, 2021.

[18] Wouter Bulten, Kimmo Kartasalo, Po-Hsuan Cameron Chen, Peter Ström, Hans Pinckaers, Kunal Nagpal, Yuannan Cai, David F Steiner, Hester Van Boven, Robert Vink, et al. Artificial intelligence for diagnosis and gleason grading of prostate cancer: the panda challenge. Nat. Med., 28(1):154–163, 2022.

[19] Andrew H Song, Richard J Chen, Tong Ding, Drew FK Williamson, Guillaume Jaume, and Faisal Mahmood. Morphological prototyping for unsupervised slide representation learning in computational pathology. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), pages 11566–11578, 2024. doi:10.1109/cvpr52733.2024.01099.

[20] Margarita L Zuley, Rose Jarosz, Bettina F Drake, Danielle Rancilio, Aleksandra Klim, Kimberly Rieger-Christ, and John Lemmerman. Radiology data from the cancer genome atlas prostate adenocarcinoma [tcga-prad] collection. Cancer Imaging Arch, 9(10.7937):K9, 2016.

[21] Geert Litjens, Peter Bandi, Babak Ehteshami Bejnordi, Oscar Geessink, Maschenka Balkenhol, Peter Bult, Altuna Halilovic, Meyke Hermsen, Rob Van de Loo, Rob Vogels, et al. 1399 h&e-stained sentinel lymph node sections of breast cancer patients: the camelyon dataset. GigaScience, 7(6):giy065, 2018. doi:10.1093/gigascience/giy065.

[22] Xitong Ling, Yuanyuan Lei, Jiawen Li, Junru Cheng, Wenting Huang, Tian Guan, Jian Guan, and Yonghong He. Comprehensive benchmark dataset for pathological lymph node metastasis in breast cancer sections. Scientific Data, 12(1):1381, 2025. doi:10.1038/s41597-025-05586-5.

[23] Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen AWM Van Der Laak, Meyke Hermsen, Quirine F Manson, Maschenka Balkenhol, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. Jama, 318(22):2199–2210, 2017. doi:10.3410/f.732283043.793567347.

[24] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Inter. Conf. Learn. Represent. (ICLR). OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.

[25] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.

[26] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: a large-scale hierarchical image database. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), pages 248–255. Ieee, 2009. doi:10.1109/CVPR.2009.5206848.

[27] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20k dataset. Inter. J. Comput. Vis., 127(3):302–321, 2019. doi:10.1007/s11263-018-1140-0.

[28] Hugo Touvron, Matthieu Cord, and Hervé Jégou. DeiT III: revenge of the ViT. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, European Conf. Comput. Vis. (ECCV), volume 13684 of Lecture Notes in Computer Science, pages 516–533. Springer, 2022. doi:10.1007/978-3-031-20053-3_30.

[29] Cijo Jose, Théo Moutakanni, Dahyun Kang, Federico Baldassarre, Timothée Darcet, Hu Xu, Daniel Li, Marc Szafraniec, Michaël Ramamonjisoa, Maxime Oquab, Oriane Siméoni, Huy V. Vo, Patrick Labatut, and Piotr Bojanowski. DINOv2 meets text: a unified framework for image- and pixel-level vision-language alignment, 2024.

[30] Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. In Adv. Neural Inf. Process. Sys. (NeurIPS), 2016.

[31] Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. The emergence of clusters in self-attention dynamics. In Adv. Neural Inf. Process. Sys. (NeurIPS), 2023.

[32] Nikita Karagodin, Yury Polyanskiy, and Philippe Rigollet. Clustering in causal attention masking. In Adv. Neural Inf. Process. Sys. (NeurIPS), 2024.

[33] Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. The unreasonable ineffectiveness of the deeper layers. In Inter. Conf. Learn. Represent. (ICLR), 2025.

[34] Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. On the effect of dropping layers of pre-trained transformer models. Comput. Speech Lang., 2023.

[35] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In Inter. Conf. Learn. Represent. (ICLR), 2020.

[36] Cristian Hinostroza, Rodrigo Toro Icarte, Christ Devia, Andres Carvallo De Ferari, Eugenio Herrera-Berg, Denis Parra, and Jorge F Silva. Rethinking layer relevance in large language models beyond cosine similarity. In Inter. Conf. Learn. Represent. (ICLR), 2026. URL https://openreview.net/forum?id=mRLnS8jQWt.

[37] Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress large language models by deleting rows and columns. In Inter. Conf. Learn. Represent. (ICLR), 2024.

[38] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-Pruner: On the structural pruning of large language models. In Adv. Neural Inf. Process. Sys. (NeurIPS), 2023.

[39] Dmitriy Shopkhoev, Ammar Ali, Magauiya Zhussip, Valentin Malykh, Stamatios Lefkimmiatis, Nikos Komodakis, and Sergey Zagoruyko. ReplaceMe: Network simplification via depth pruning and transformer block linearization. In Adv. Neural Inf. Process. Sys. (NeurIPS), 2025. URL https://openreview.net/forum?id=zEj1FSYCRn.

[40] Sayantan Dasgupta and Trevor Cohn. Improving language model distillation through hidden state matching. In Inter. Conf. Learn. Represent. (ICLR), 2025.

[41] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Adv. Neural Inf. Process. Sys. (NeurIPS), 2020.

[42] Malthe Have Musaeus and Rob van der Goot. Iterative structured knowledge distillation: Optimizing language models through layer-by-layer distillation. In Inter. Conf. Comput. Ling. (COLING), 2025.

[43] Nobuyuki Otsu et al. A threshold selection method from gray-level histograms. Automatica, 11(285-296):23–27, 1975. doi:10.1109/tsmc.1979.4310076.