The Dead Salmons of AI Interpretability cover

📄 Executive Summary

Artificial intelligence interpretability methods frequently suffer from severe statistical fragility, producing false discoveries analogous to the famous neuroscience demonstration where standard functional MRI procedures detected brain activity in a dead salmon. Common interpretability techniques—including feature attribution, probing classifiers, sparse autoencoders, circuit discovery, and natural language explanations—often generate plausible explanations for completely random or untrained neural networks. In real-world deployment, these methods display high variance, extreme sensitivity to minor perturbations or design choices, and a failure to generalize to new data. The primary conclusion of this work is that these widespread failures stem from a fundamental theoretical problem: non-identifiability, where the internal traces of a computational network do not uniquely determine a single explanation. To resolve this, the authors argue that interpretability must be reframed as a formal problem of statistical and causal inference.

To diagnose these issues, the authors surveyed documented failure modes across major interpretability families and traced their underlying causes. Purely behavioral and predictive methods suffer from underspecification, in which many competing explanations fit observed data equally well. Meanwhile, causal and mechanistic methods face overdetermination, where complex neural networks possess multiple redundant, independently sufficient computational pathways. Under these conditions, standard interpretability queries become mathematically non-identifiable. Consequently, different analysis pipelines arbitrary select different explanations from an unidentifiable set, causing high instability, false discoveries, and inflated confidence.

To establish methodological guardrails, the authors propose a statistical-causal framework that treats explanations not as single objective truths, but as surrogate models inferred from finite computational traces. In this formulation, an interpretability task is explicitly defined by three components: a distribution of causal queries representing what needs explaining, a class of admissible surrogate explanations, and a discrepancy measure quantifying error. Interpretability methods act as statistical estimators mapping sampled computational traces to estimated surrogate models. This structure allows researchers to formulate clear null hypotheses, evaluate the identifiability of their questions, and report quantified uncertainty, such as confidence sets or Bayesian posterior distributions, rather than single point estimates.

As an immediate intervention, the authors demonstrate a hypothesis-testing protocol that evaluates explanations from trained networks against null baselines derived from randomized network weights. In experiments using BERT and Pythia models across sentiment analysis, part-of-speech tagging, and geospatial representation tasks, testing against randomized nulls eliminated spurious findings and significantly reduced inflated effect sizes. For instance, sentiment probes that appeared highly significant under standard analyses were indistinguishable from random reinitializations, whereas genuine representations in syntactic and world-model tasks remained statistically detectable.

These findings matter because AI systems are increasingly deployed in high-stakes environments where interpretability is necessary for safety, accountability, and error mitigation. Relying on fragile methods creates a false sense of security. However, the authors note several open questions and limitations. Identifying non-identifiability does not immediately resolve it; practical trade-offs remain between query expressivity, sample efficiency, and human cognitive capacity to understand complex explanations. The authors recommend that future work focus on formalizing identifiability conditions across common tasks, adopting Bayesian methods for uncertainty estimation, pre-registering analyses, and establishing cumulative meta-analyses to place AI interpretability on a rigorous empirical foundation.

📚 What to read before this

The Mythos of Model Interpretability, Zachary C. Lipton (2016). Lipton establishes the foundational critique of the ambiguous use of "interpretability" in machine learning, providing the precise conceptual backdrop that the source adapts to argue for treating explanations as statistical model parameters.
Sanity Checks for Saliency Maps, Julius Adebayo et al. (2018). Adebayo et al. demonstrate that standard saliency methods yield false discoveries on randomized neural networks, supplying the exact empirical "dead salmon" phenomenon of illusory AI explanations that motivates the source's statistical-causal reframing.
Shortcut learning in deep neural networks, Robert Geirhos et al. (2020). Geirhos et al. analyze shortcut learning and spurious feature reliance in neural networks, offering the core empirical evidence of misleading interpretability signals that the source addresses through rigorous alternative hypothesis testing.

📚 What to read after this

Scaling and evaluating sparse autoencoders, Leo Gao et al. (2025). Gao et al. extend the pursuit of faithful feature extraction by scaling and evaluating sparse autoencoders, confronting the high variance and dead-latent challenges that directly follow from the source's critique of interpretability artifact generation.
Identifying Weight-Variant Latent Causal Models, Yuhang Liu et al. (2026). Liu et al. build directly upon the source's causal-statistical paradigm by deriving identifiability guarantees and learning methods for latent causal models under varying generative mechanisms.

The Dead Salmons of AI Interpretability

Maxime Méloux$^{\diamondsuit}$, Giada Dirupo$^{\clubsuit}$, François Portet$^{\diamondsuit}$, Maxime Peyrard$^{\diamondsuit}$

$^{\diamondsuit}$ Université Grenoble Alpes, CNRS, Grenoble INP, LIG $^{\clubsuit}$ Icahn School of Medicine at Mount Sinai

{melouxm,peyrardm}@univ-grenoble-alpes.fr

Abstract

In a striking neuroscience study, the authors placed a dead salmon in an MRI scanner and showed it images of humans in social situations. Astonishingly, standard analyses of the time reported brain regions predictive of social emotions. The explanation, of course, was not supernatural cognition but a cautionary tale about misapplied statistical inference. In AI interpretability, reports of similar "dead salmon" artifacts abound: feature attribution, probing, sparse auto-encoding, and even causal analyses can produce plausible-looking explanations for randomly initialized neural networks. In this work, we examine this phenomenon and argue for a pragmatic statistical–causal reframing: explanations of computational systems should be treated as parameters of a (statistical) model, inferred from computational traces. This perspective goes beyond simply measuring statistical variability of explanations due to finite sampling of input data; interpretability methods become statistical estimators, and findings should be tested against explicit and meaningful alternative computational hypotheses, with uncertainty quantified with respect to the postulated statistical model. It also highlights important theoretical issues, such as the identifiability of common interpretability queries, which we argue is critical to understand the field’s susceptibility to false discoveries, poor generalizability, and high variance. More broadly, situating interpretability within the standard toolkit of statistical inference opens promising avenues for future work aimed at turning AI interpretability into a pragmatic and rigorous science.

Executive Summary: Artificial intelligence interpretability methods frequently suffer from severe statistical fragility, producing false discoveries analogous to the famous neuroscience demonstration where standard functional MRI procedures detected brain activity in a dead salmon. Common interpretability techniques—including feature attribution, probing classifiers, sparse autoencoders, circuit discovery, and natural language explanations—often generate plausible explanations for completely random or untrained neural networks. In real-world deployment, these methods display high variance, extreme sensitivity to minor perturbations or design choices, and a failure to generalize to new data. The primary conclusion of this work is that these widespread failures stem from a fundamental theoretical problem: non-identifiability, where the internal traces of a computational network do not uniquely determine a single explanation. To resolve this, the authors argue that interpretability must be reframed as a formal problem of statistical and causal inference.

To diagnose these issues, the authors surveyed documented failure modes across major interpretability families and traced their underlying causes. Purely behavioral and predictive methods suffer from underspecification, in which many competing explanations fit observed data equally well. Meanwhile, causal and mechanistic methods face overdetermination, where complex neural networks possess multiple redundant, independently sufficient computational pathways. Under these conditions, standard interpretability queries become mathematically non-identifiable. Consequently, different analysis pipelines arbitrary select different explanations from an unidentifiable set, causing high instability, false discoveries, and inflated confidence.

To establish methodological guardrails, the authors propose a statistical-causal framework that treats explanations not as single objective truths, but as surrogate models inferred from finite computational traces. In this formulation, an interpretability task is explicitly defined by three components: a distribution of causal queries representing what needs explaining, a class of admissible surrogate explanations, and a discrepancy measure quantifying error. Interpretability methods act as statistical estimators mapping sampled computational traces to estimated surrogate models. This structure allows researchers to formulate clear null hypotheses, evaluate the identifiability of their questions, and report quantified uncertainty, such as confidence sets or Bayesian posterior distributions, rather than single point estimates.

As an immediate intervention, the authors demonstrate a hypothesis-testing protocol that evaluates explanations from trained networks against null baselines derived from randomized network weights. In experiments using BERT and Pythia models across sentiment analysis, part-of-speech tagging, and geospatial representation tasks, testing against randomized nulls eliminated spurious findings and significantly reduced inflated effect sizes. For instance, sentiment probes that appeared highly significant under standard analyses were indistinguishable from random reinitializations, whereas genuine representations in syntactic and world-model tasks remained statistically detectable.

These findings matter because AI systems are increasingly deployed in high-stakes environments where interpretability is necessary for safety, accountability, and error mitigation. Relying on fragile methods creates a false sense of security. However, the authors note several open questions and limitations. Identifying non-identifiability does not immediately resolve it; practical trade-offs remain between query expressivity, sample efficiency, and human cognitive capacity to understand complex explanations. The authors recommend that future work focus on formalizing identifiability conditions across common tasks, adopting Bayesian methods for uncertainty estimation, pre-registering analyses, and establishing cumulative meta-analyses to place AI interpretability on a rigorous empirical foundation.

1. Introduction

Section Summary: The 2009 “dead salmon” experiment exposed how uncorrected statistical errors in brain imaging could produce convincing but meaningless results, sparking broader reforms in neuroscience to address false positives, small samples, and fragile analyses. AI interpretability now shows similar flaws, with popular explanation techniques generating apparently meaningful insights even when applied to completely random neural networks, and these outputs often vary wildly with minor input changes or different random seeds. The paper argues that these problems stem from deeper issues of non-identifiability and calls for pragmatic, goal-oriented standards rather than claims of uncovering a single true internal mechanism.

In 2009, researchers placed a dead salmon in an MRI scanner, showed it photographs of humans in social situations, and ostensibly asked it to judge their emotions ([1]). Standard analysis pipelines commonly used at the time surprisingly returned brain voxels as significantly predictive of emotional situations. The error arose from a failure to correct for multiple comparisons within the statistical analysis pipeline. The “dead salmon” demonstration of false positives contributed to a larger reckoning in the field of neuroscience. For instance, an influential study showed that different research groups obtained different results even when analyzing the same dataset and the same research question ([2]). Subsequent work identified several sources of statistical fragility. Widely used statistical procedures embedded in standard analysis pipelines were shown to inflate false-positive rates ([3]), an effect worsened by non-independent analyses producing spuriously large brain–behavior correlations ([4]). Also, early neuroimaging research was constrained by small samples and limited data availability ([5, 6]), exacerbating overfitting and spurious associations. Moreover, fMRI had been criticized for offering predictive explanations, rather than functional ones, resulting in little clinical relevance ([7]). Finally, reverse inference emerged as a central interpretative problem, given that individual neural systems are not uniquely associated with specific cognitive functions ([8, 9]).

AI interpretability now faces its own dead salmon issues, similarly begging for a larger reevaluation of its statistical foundations. A growing body of work has shown that many influential methods, including feature attribution ([10]), probing classifiers ([11]), sparse autoencoders ([12]), circuit discoveries ([13]), and causal abstractions ([14]), can yield plausible-looking explanations even when applied to random neural networks. In Figure 1, we report a minimum dead salmon artifact from analyzing activations of a fully randomized BERT model in a sentiment analysis task where both correlation analysis and probing find highly significant explanations. Such striking failure modes are particularly troubling as modern AI systems are increasingly deployed in high-stakes domains where AI interpretability should be essential for transparency, accountability, and error diagnosis ([15, 16, 17]). Interpretability methods have the potential surface critical failure modes ([18, 19, 20, 21, 22, 23]) and offer levers for mitigating bias and systematic errors ([24, 25, 26]).

Yet, despite frequent analogies to mature sciences like neuroscience ([27]), biology ([28]), or physics ([29, 30]) of neural networks, the practice of AI interpretability remains in its early foundational stages. Striking dead-salmon artifacts are accompanied by a general statistical fragility: small perturbations to inputs ([31, 32, 33]) or changes in random initialization ([10, 34]) can radically change explanations. Explanations often fail to generalize to new settings and input distributions ([35]). Also, multiple incompatible explanations can be discovered for the same behavior ([13, 36]). While the dead salmon study demonstrated a simple statistical oversight correctable through multiple comparison adjustments, AI interpretability's difficulties stem from more fundamental issues. In particular, we argue that, for common interpretability queries, computational traces do not uniquely determine explanations.

**Figure 1:** **Minimal dead salmon artifacts**. We extract token representations from a randomly initialized BERT for 300 IMDb sentences and average over sequence length. (A) Several principal components are spuriously correlated with sentiment labels. (B) A simple probe trained on layer representations achieves nontrivial cross-validated accuracy.

Beyond neuroscience and AI, such challenges are not unprecedented. Psychology and the social sciences faced a similar reckoning during the replication crisis, when questionable research practices produced widespread false positives ([37, 38, 39, 40]). These fields responded with methodological reforms: pre-registration, registered reports, increased statistical power, and explicit multiple-comparison corrections ([41, 42]). Likewise, econometrics used causal inference ([43]) to formalize the distinction between correlation and causation, developing identification criteria, sensitivity analyses, and robustness tests ([44, 45, 46]).

Now, AI interpretability can also begin to build its own methodological guardrails. As argued before, this requires both technical innovation and philosophical clarity ([47, 48]). This means clarifying our epistemic goals by answering: what does it mean to "explain" a neural network? ([49, 50]) Mechanistic interpretability embodies a type of scientific realism, aiming to discover the one true explanatory algorithm ([51, 52]). However, there is a significant push-back against the feasibility of this research project ([53, 54, 55]), motivating a shift toward pragmatic approaches prioritizing the utility of the explanations for specific downstream goals ([56]). Here, we align with the pragmatic stance ([57, 58, 59]), where explanations are seen as useful models that enable prediction, manipulation, and control ([60, 61]).

This work. We analyze failure modes of contemporary AI interpretability methods, ranging from striking dead-salmon false positives to broader forms of statistical fragility, including poor generalization and high variance. We argue that these pathologies share a common root cause: the non-identifiability of many interpretability queries, compounded by the lack of principled uncertainty quantification, where non-identifiability manifests as high-variance estimates that should be reflected in large uncertainty. Diagnosing and addressing these issues, as well as articulating a coherent pragmatic research direction for interpretability, requires reframing AI interpretability as a problem of statistical (causal) inference. Accordingly, we propose one such statistical–causal reframing in which explanations are treated as parameters inferred from computational traces, enabling uncertainty-aware evaluation against meaningful alternative computational hypotheses.

2. The Statistical Fragility of AI Interpretability

Section Summary: Many AI interpretability methods prove statistically fragile because they can generate plausible-looking results even when applied to randomized models or irrelevant data. Techniques such as gradient-based attributions, probing classifiers, sparse autoencoders, and causal interventions often produce explanations that fail to generalize, remain vulnerable to small changes or adversarial tweaks, and create illusions of insight without revealing what the model actually uses. Similar weaknesses affect concept-based approaches and natural-language rationales, which readily offer convincing but unfaithful stories about model behavior.

Reports documenting the failure modes of interpretability methods are frequent and highlight a recurring theme: a general statistical fragility, most strikingly illustrated by dead salmon artifacts. We provide here a non-exhaustive overview of such issues.

Feature Attribution. Gradient-based attribution methods ([62, 63]) aim to highlight input features most relevant to model predictions. However, [10] demonstrated that saliency maps can remain visually plausible even after model weights are randomized. Further, [36] showed that gradient-based explanations can be manipulated by adversarial perturbations, leaving predictions unchanged, while [31] revealed that explanations are unstable under minor data transformations. From a theoretical standpoint, [64] established impossibility results showing that no attribution method can simultaneously satisfy intuitive desiderata across broad model classes.

Probing. Probing methods train a classifier to predict a target label from internal activations. Early studies already showed that both linear and structural probes could recover information with surprisingly high accuracy from randomized contextualized embeddings ([65, 66]), and syntactic probes do not generalize ([67]). Later, [11] demonstrated that probes can extract features merely encoded (e.g., inherited from embeddings) even if unused during inference; probing asks whether a concept is encoded in an activation, not whether it is computationally relevant. Capacity-controlled and information-theoretic probes ([68, 69, 70, 71]) or amnesic probing ([72]) attempt to mitigate such false discoveries.

Sparse Autoencoders. Unsupervised concept-discovery pipelines such as sparse autoencoders (SAEs) ([73, 74, 75, 76]) display analogous pathologies. [12] showed that SAEs can recover apparently interpretable components even in randomly initialized transformers. Additional studies show that SAEs often fail to generalize across settings or tasks ([77, 78]). Also, [79] show SAE are sensitive to adversarial input perturbations.

Concept-Based Explanations. Concept-based methods ([80, 81]) aim to identify human-interpretable concepts that align with model representations (e.g., concept activation vectors ([80]) or network dissection ([81])). These methods also face documented limitations ([82, 83]). Already, [84] showed interpretability illusion arising where activations of individual neurons in BERT may spuriously appear to encode a concept. Then, [85] showed that concept activation scores can produce highly inconsistent explanations, and [86] documented poor generalization and high sensitivity to the dataset used to infer concepts. Finally, [87] further demonstrated high variance and recommended incorporating uncertainty estimation.

Causal Approaches. To address issues with prediction-based explanations, a shift toward causality-based interpretability has emerged through the use of causal mediation analysis ([88, 72, 89, 21, 90, 91, 22, 92]). These methods intervene on intermediate representations to quantify causal effects of components on model outputs. Yet recent work documents substantial fragilities and trade-offs ([93]): [94] showed that such approaches are sensitive to experimental design. Then, [95] discovered a "hydra effect, " where ablating components identified as causally important fail to change behavior due to redundant causal pathways. This phenomenon, known as overdetermination, occurs when multiple redundant, independently sufficient causal pathways exist ([96, 97, 98]). Rather than isolating simple mechanisms, interventions tend to reveal overdetermined causal structures.

Mechanistic Interpretability. Causal approaches culminate in mechanistic interpretability (MI), which aims to reverse-engineer networks into human-interpretable algorithms ([99]). One family of approaches (where-then-what) first identifies circuits carrying information from inputs to outputs and then interprets their components ([100, 101, 102]). The second (what-then-where) instead starts from high-level candidate algorithms and searches for causally aligned neural subspaces, using causal abstraction metrics ([103, 104, 105]). Despite promising demonstrations, both categories have the typical issues ([106]). Subspace patching can produce interpretability illusions by activating alternate pathways ([107]), also a problem of overdetermination. Circuit explanations often fail to generalize ([108, 109]) and are sensitive to minor experimental choices ([110]). Exhaustive studies on toy models reveal multiple incompatible explanations for both strategies, even for random networks ([13]). Finally, [14] proved that, in general, existing causal abstraction methods can produce explanations for random networks.

Natural Language Explanations. Generating natural language rationales has become a popular interpretability approach ([111, 112]). However, [113] showed that LLM-generated explanations can be systematically unfaithful, confidently providing plausible-sounding justifications for predictions made for entirely different reasons. Moreover, chain-of-thought (self-)explanations are typically unfaithful to the model’s computation ([114, 115, 116]). There exist infinite plausible stories that can rationalize any behavior post hoc. Thus, natural language explanations are particularly susceptible to confabulation and, thus, to false positives.

3. The Deeper Statistical Issue

Section Summary: The core statistical weakness in AI interpretability is non-identifiability: many distinct explanations can fit the same observed behavior equally well. Behavioral methods suffer from underspecification, where multiple accounts match input-output patterns; internal predictive techniques inherit overfitting and poor generalization; and causal approaches run into overdetermination, as complex models contain redundant pathways that each suffice to produce the same outputs. The result is explanations that generalize unreliably, shift with arbitrary design choices, and carry a high risk of spurious findings.

The problems documented in Section 2 point to a broad statistical fragility. Here, we identify the common structure underlying these failures: the non-identifiability of interpretability queries.

Behavior-based approaches that study input–output relationships (e.g., feature attributions, behavioral testing) are fundamentally limited by underspecification: multiple, distinct explanations can equally well account for the same input–output patterns ([117, 118, 119]). Similar observations in cognitive science motivated the development of brain imaging as a complement to purely behavioral data, with the goal of measuring neural computation and thereby obtaining objective, measurable, and more generalizable quantities ([120, 121, 122]). AI interpretability has followed a related trajectory moving toward analyzing internal computation ([92]). However, predictive approaches based on internal states (probing, SAEs) inherit standard machine-learning pathologies such as overfitting and poor generalization ([71]). These failure modes are instances of underspecification: many predictive models can fit the training data equally well, leaving it unclear which ones posit generalizable causal mechanisms ([123, 124]).

Causal approaches, introduced in response to the shortcomings of predictive methods, appear at first to provide the scientific rigor needed for generalizable explanations. However, AI systems are large, distributed systems with many interacting components, which gives rise to redundant and context-dependent causal pathways ([125]). This creates overdetermination, where multiple distinct causal mechanisms are each independently sufficient to produce the same behavior ([126, 127, 128]). Then, finding mechanistic stories within complex computational systems can become too easy: many different, incompatible explanations can be produced for the same phenomenon ([129, 13]).

Identifiability. These failure modes can be formalized using the concept of identifiability. Informally, identifiability is the property of a statistical inference task stating that the parameters (explanatory variables) of a statistical model can be uniquely recovered from available observations ([130]). Identifiability is typically a prerequisite for reliable inference in the natural sciences; without it, inferred explanations remain ambiguous. Therefore, substantial work in statistics, unsupervised learning, and causal inference has focused on characterizing identifiability conditions and designing identifiable tasks ([130, 131, 132, 133, 134]).

For interpretability, both underspecification and overdetermination produce non-identifiability, explaining most of the statistical fragilities: (i)

Poor generalization: when multiple explanations fit the observed data equally well, their explanatory claims can diverge arbitrarily on unseen data. Selecting among these explanations, therefore, depends on arbitrary inductive biases that are rarely validated. (ii)

Sensitivity to design choices: non-identifiability implies a manifold of explanations that achieve a good fit. Different algorithmic choices (datasets, optimization procedures, hyperparameters) traverse this manifold differently, and thus produce different explanations. (iii)

False discovery: when explanations are non-identifiable, the probability of recovering a spurious explanation that happens to fit the data increases with the size and complexity of the hypothesis space.

Currently, identifiability is just a conceptual analogy, because interpretability has not yet been formalized as an explicit statistical inference task. Making this formal connection and casting interpretability queries as well-specified statistical estimation problems is a necessary first step toward developing methods whose limitations and assumptions can be explicitly characterized.

4. The Statistical–Causal Inference Perspective

Section Summary: The section proposes moving beyond simple randomization tests—which compare interpretability results on trained networks against networks with random weights to filter out artifacts—and instead reframing the entire field as a problem of statistical-causal inference. In this view, explanations function as surrogate models that answer causal queries about how a system's internal components influence its observed behavior under interventions. The authors ground this perspective in standard tools from statistics and causal modeling, such as identifiable structural causal models, to ensure explanations support reliable prediction, manipulation, and generalization.

A straightforward way to address dead-salmon artifacts across interpretability methods is to compare findings on a trained target network against a randomized alternative: the same architecture with randomized weights analyzed by the same method. This leads to a principled hypothesis test against a null hypothesis of randomized computation, an idea foreshadowed in early work on probing ([65, 66, 11]) and circuit discovery ([135]). We formalize such a test in Appendix A and show that, for probing, it eliminates some false discoveries and substantially reduces effect sizes in standard analyses.

While effective, directly correcting dead-salmon artifacts is a very low bar for interpretability. The goal is to address the deeper statistical issues that give rise to these failures in the first place. Nevertheless, hypothesis testing against computationally meaningful null alternatives naturally motivates a broader statistical–causal reframing of interpretability.

Here, we sketch one such formalization, viewing interpretability as a problem of statistical–causal inference. In this view, explanations are surrogate models constructed to answer distributions of causal queries about a computational system. An explanation is useful insofar as it supports prediction and manipulation, generalizes under intervention, and remains robust to noise. This perspective aligns with a growing pragmatist approach to interpretability ([54]).

4.1 Background: Statistical–Causal Inference

Statistical inference provides the rigorous framework through which empirical observations become scientific knowledge ([136, 137]). We argue that interpretability, like every empirical science, must be grounded in these principles. We provide here a brief overview.

Statistical Models and Identifiability.

A statistical model is a family of probability laws ${\mathbb{P}\theta^{\mathbf{V}}:\theta\in\Theta}$ on a sample space $\mathcal{V}$, indexed by parameters $\theta\in\Theta$. Here, $\mathbf{V}$ denotes observed data. Intuitively, we assume data arises from some process indexed by unknown parameters $\theta$, and the goal is to recover $\theta$ from observations. Sound inference requires identifiability: distinct parameters must induce distinct distributions over observables. Formally, a model is identifiable if $\theta \neq \theta' \implies \mathbb{P}\theta^{\mathbf{V}} \neq \mathbb{P}_{\theta'}^{\mathbf{V}}$. Without identifiability, hypotheses cannot be distinguished from data, rendering inference ill-posed.

Estimators and Uncertainty Quantification.

Given finite observations $\mathcal{D}n={\mathbf{v}^{(i)}}{i=1}^n$, an estimator $T$ produces an estimate $\hat{\theta} := T(\mathcal{D}_n)$ of unknown parameters $\theta$. Its quality can be assessed through various statistical properties: (i) Bias: Does it recover the correct parameter on average? (ii) Variance: How much does the estimate vary across datasets? (iii) Consistency: Does it converge to the correct parameter as $n\to\infty$? Beyond point estimates, confidence sets provide uncertainty quantification under finite sampling.

Causal Inference.

Many scientific questions go beyond prediction, seeking explanations of how variables influence one another. This requires enriching statistical models with a causal structure ([43]). Let $\mathbf{V}={V_1, \ldots, V_d}$ denote endogenous variables, quantities computed within the system. A directed graph $\mathcal{G}$ over nodes $\mathbf{V}$ encodes direct causal relationships: an edge $V_i \to V_j$ indicates that $V_i$ directly causes $V_j$. A structural causal model (SCM) is the tuple $\mathfrak{C}=(\mathbf{V}, \mathbf{U}, \mathbf{f}, P_{\mathbf{U}})$, where:

$\mathbf{U}$ collects exogenous (external) inputs representing unobserved causes or environmental randomness, $P_{\mathbf{U}}$ is their joint distribution
$\mathbf{f}={f_1, \ldots, f_d}$ are structural assignments, functions that deterministically compute each variable from its causes:

$ V_i = f_i(\mathbf{PA}_i, U_i), \qquad i=1, \ldots, d,\tag{1} $

where $\mathbf{PA}_i\subseteq \mathbf{V}$ denotes the parents of $V_i$ in $\mathcal{G}$, and $U_i\in \mathbf{U}$ is its exogenous input.

$**Figure 2:** The tuple of a target behavior $P_{\mathbf{U}}$ and the computational system with its internal components form an SCM.$

SCMs enable reasoning about interventions and counterfactuals. A hard intervention $\mathrm{do}(\mathbf{V}_I= \mathbf{v}_I)$ on a subset $\mathbf{V}_I\subseteq \mathbf{V}$ replaces the structural assignments for variables in $\mathbf{V}_I$ with constants $\mathbf{v}_I$, overriding their causal mechanisms. The intervened model $\mathfrak{C};\mathrm{do}(\mathbf{V}_I= \mathbf{v}_I)$ induces an interventional distribution $P^{\mathfrak{C};\mathrm{do}(\mathbf{V}_I= \mathbf{v}I)}{\mathbf{V}}$, which captures how the system behaves under this external manipulation.

A causal query $q(\mathfrak{C})$ is any well-defined question about the SCM, such as "What is the marginal distribution of $V_i$?" or "What is the average effect of setting $V_i=v$ on outcome $V_m$?" Thus, a causal query is any measurable functional of the SCM, possibly involving conditioning or intervention. Central to causal inference is query identifiability: whether $q(\mathfrak{C})$ can be uniquely determined from available observational or interventional data.

4.2 Neural Networks as Structural Causal Models

Returning to modern AI interpretability, we first state a standard framing of computational systems as SCMs. Let $f$ be a computational system, typically a neural network, with internal computational elements $\mathbf{V}$ and input distribution $P_{\mathbf{U}}$. The input distribution $P_{\mathbf{U}}$ represents the behavior of interest that we aim to explain. For instance, $P_{\mathbf{U}}$ might represent arithmetic prompts to a language model, images from a particular domain, or factual questions about a specific topic.

The tuple $(f, P_{\mathbf{U}})$ naturally defines an SCM $\mathfrak{C} = (\mathcal{G}, ; \mathbf{V}, ; \mathbf{U}, ; \mathbf{f}, ; P_{\mathbf{U}})$, where:

Endogenous variables $\mathbf{V}$ are the network's computational variables (e.g., hidden states, attention patterns, outputs).
Exogenous variables $\mathbf{U}$ are inputs sampled from $P_{\mathbf{U}}$, representing the behavior we seek to explain.
Structural assignments $\mathbf{f}$ are the deterministic functions defining the network's computation (layers, attention mechanisms, nonlinearities).
Causal graph $\mathcal{G}$: the network's computation graph.

The SCM induces a unique observational distribution over $\mathbf{V}$: sampling corresponds to drawing inputs from $P_{\mathbf{U}}$, executing a forward pass, and recording desired activations. Also, the SCM encodes interventional and counterfactual distributions based on external modifications of the inner computation. This perspective is standard within mechanistic interpretability ([138, 139, 104, 140]) and is illustrated in Figure 2. Then, a causal query is any well-specified quantity about $\mathfrak{C} := (f, P_{\mathbf{U}})$, such as "What distribution would the network produce if we forced activation $V_i$ to value $v$?" or "How much does attention head $V_a$ causally contribute to correct factual recall?"

4.3 Explanations as Surrogate Models

In an attempt to provide a general statistical-causal perspective on interpretability, we formalize explanations as surrogate models: simpler computational descriptions designed to answer chosen collections of causal queries about a target system. This perspective treats interpretability as a form of model compression, where we seek a simpler model that faithfully approximates a complex system's behavior for queries we care about. In this perspective, every interpretability method is characterized by three ingredients:

Query space $Q$ with distribution $\mu$: The set of causal queries to be answered by the explanation. It dictates what aspects of $\mathfrak{C}$ should be explained. This encodes our explanatory goals.
Surrogate class $\mathcal{E}$: The class of admissible explanations. It dictates what forms the explanation can take, e.g., circuits, sparse subgraphs, linear probes, concept vectors, causal graphs, …
Discrepancy measure $D$: How we measure whether a surrogate (member of $\mathcal{E}$) correctly answers queries.

This framework is (non-rigorously) illustrated with the example of circuit discovery in Figure 3. While standard causal inference often concentrates $\mu$ on a single causal query (e.g., average treatment effect), interpretability aims to answer many diverse queries drawn from a non-trivial distribution $\mu$. For example, $\mu$ might distribute probability over interventional queries and counterfactual queries across network components, or any functionals of the interventional and counterfactual distributions.

For instance, we can view each candidate explanation $e \in \mathcal{E}$ as defining a query-answering map $S_e: Q \to \mathcal{R}$, where $S_e(q)$ is the surrogate's predicted answer to query $q$, and $\mathcal{R}$ is the space of possible answers for query $q$ (e.g., probability distributions, scalar effects, or discrete predictions). The surrogate's fidelity is measured by the discrepancy function $D: \mathcal{R} \times \mathcal{R} \to \mathbb{R}_+$ quantifying the error between the answer from $q(\mathfrak{C})$ and the surrogate's prediction $S_e(q)$. Then, we can define the population risk of a candidate explanation as the expected error over queries:

$ L_\mu(e) = \mathbb{E}_{q\sim\mu}\big[D\big(q(\mathfrak{C}), , S_e(q)\big)\big].\tag{2} $

An ideal explanation $e^* \in \mathcal{E}$ minimizes this risk: $e^* \in \arg\min_{e \in \mathcal{E}} L_\mu(e)$.

$**Figure 3:** An interpretability task is defined by three elements: $\mathcal{E}$, the hypothesis space; $\mu$, the distribution over causal queries about the SCM (model and behavior); and $D$, the error measure.$

Interpretability Task and Identifiability.

We call the triple $(\mu, \mathcal{E}, D)$ an interpretability task, fully specifying what we aim to explain (query distribution $\mu$), what explanations are admissible (hypothesis class $\mathcal{E}$), and how we measure success (discrepancy $D$). The task is identifiable if $L_\mu$ admits a unique minimizer in $\mathcal{E}$ (potentially up to predefined acceptable symmetries). Identifiability captures whether the surrogate class can, in principle, be distinguished using the queries deemed relevant by $\mu$. Without identifiability, multiple incompatible explanations achieve the same population risk, making inference fundamentally ambiguous. The analysis of Section 2 indicates that the most common tasks are not identifiable. Finally, this reframing highlights that explanations are pragmatic computational summaries of the structure encoded by $\mathfrak{C} := (f, P_{\mathbf{U}})$. They are inferred models useful for specific explanatory purposes.

Estimation with Finite Data.

In practice, we face two types of finite sampling difficulties. First, we observe only finitely many queries $q_1, \ldots, q_n \sim \mu$ from the query distribution. Second, for each query $q_j$, we can only collect a finite amount of computational traces by sampling inputs $\mathbf{U} \sim P_{\mathbf{U}}$ and recording the corresponding activations and outputs, potentially under interventions. Let $\mathcal{T}_n$ denote the complete dataset of query-trace pairs. An interpretability method $M$ acts as an estimator, mapping this finite dataset to an estimated surrogate explanation:

$ \hat{e} := M(\mathcal{T}_n).\tag{3} $

A natural estimator is given by empirical risk minimization: choose $\hat{e}$ to minimize the empirical risk $\widehat{L_\mu}(e) = \frac{1}{n}\sum_{j=1}^n \widehat{D}(q_j(\mathfrak{C}), , S_e(q_j))$, where $\widehat{D}$ is estimated from finite traces.

Relevant to this exposition, [141] proposed a statistical framing for feature attribution. Then, previous works already explored hypothesis testing and uncertainty quantification for circuit discovery ([135, 110]).

4.4 Re-Interpreting Documented Issues

The framework does not prescribe which $(\mu, \mathcal{E}, D)$ researchers should adopt. Rather, it provides a shared language for making assumptions explicit and rooted in the tools of statistical inference. Different research programs will choose different hypothesis classes or query distributions; the framework ensures that such choices are transparent and their implications are analyzable. Table 1 in the appendix illustrates how existing interpretability methods can be mapped into this formulation, each implicitly making assumptions about queries, surrogates, and error metrics. Under this view, the issues described in Section 2 can be understood as problems of non-identifiability.

Behavioral Benchmarks.

Benchmarks that evaluate model outputs against a gold standard (averaged success over input distributions) ask an identifiable question: how well does the model perform under a specific task distribution and error metric? This is arguably the simplest form of interpretability and drove most of the progress in AI. Its usefulness depends on the construction of the benchmark, but the inference problem is well-posed.

Concept-Based Approaches.

Predictive methods (probes, SAEs) inherit non-identifiability issues from the underlying underspecification of machine learning tasks ([124]). For example, methods like Concept Activation Vectors ([80, 73]) postulate that internal states $\mathbf{v}$ are generated by interpretable concepts $\mathbf{z}$ via $\mathbf{v} = g(\mathbf{z})$. This is an instance of (causal) representation learning, which is non-identifiable without auxiliary information ([132, 133]). It is therefore unsurprising that proposed improvements mirror standard remedies for underspecification in machine learning: regularization in the form of capacity control for probes ([71]) or cross-validation to assess generalization ([78]).

Causal Mediation Analysis.

Methods like causal mediation analysis ([21, 142]) estimate the indirect effect of a component on observed outputs. As the intervention and model are fully specified, the mediation estimand is unique and identifiable. However, the explanatory claim that a component with high effect is the locus of a mechanism is not identifiable, because of the overdetermined causal structure.

Circuit Discovery and Causal Approaches.

Circuit discovery seeks a subgraph $G' \subset G$ that preserves the model's performance. This task faces the "Hydra effect" ([95]) and causal overdetermination. If parallel pathways $A$ and $B$ are sufficient, circuits containing only $A$ or only $B$ both satisfy fidelity criteria. Thus, even correct causal methods may recover many different explanations consistent with the same behavior ([13]). Addressing this requires formulating identifiable causal questions. Causal abstraction ([105, 140]) offers a promising direction, as it operates at a coarser representational level where overdetermination can be absorbed into the abstracted representations. However, current operational metrics demonstrate empirical non-identifiability ([13, 14]).

5. Discussion

Section Summary: The discussion traces many failures in neural network interpretability to non-identifiability, where available data cannot uniquely determine the intended explanation. It therefore advocates recasting interpretability as a statistical-causal inference problem that separates the target of explanation from the form of the explanation, treats the task as an objective inference once those choices are fixed, and requires explicit uncertainty quantification rather than point estimates alone. The section further sketches practical steps for making such tasks more identifiable and outlines future work on characterizing identifiability conditions and Bayesian approaches.

The systematic failures documented in Section 2 demanded an explanation. We have argued that these pathologies share a common root cause: non-identifiability. Most current interpretability tasks attempt to infer explanatory structures that are not uniquely determined by available computational traces. To trace a path forward, we proposed one formalization of interpretability as statistical-causal inference. This framework is tentative rather than definitive; we encourage the community to improve upon it. The important aspect is the methodological commitment to making assumptions explicit and quantifying uncertainty rigorously.

5.1 Advantages of the Statistical-Causal Perspective

Drawing on the philosophy of science ([58, 143, 59]) and recent calls for a pragmatic approach to interpretability ([101, 48]), the framework naturally distinguishes the explanandum (what is to be explained, encoded in $\mu$) from the explanans (what does the explaining, encoded in $\mathcal{E}$). Researchers and practitioners have substantial freedom in choosing both. There is no single "correct" explanation of a neural network. The appropriate type of description depends on one's purposes ([59]). Descriptive understanding corresponds to queries about observational distributions; predictive goals involve queries requiring surrogates to generalize to new input distributions; control and intervention require queries about counterfactual or interventional distributions.

However, once the explanatory project is specified, i.e., once $(\mu, \mathcal{E}, D)$ are fixed, the explanation becomes an objective inference problem. The best surrogate $e^* \in \mathcal{E}$ is the one minimizing $L_\mu(e)$, and is a property of the system itself and the interpretability task. If the task is identifiable, this explanation is unique (up to permissible symmetries, e.g., rotation invariance in representation space). This reconciles pluralism about explanatory goals with rigor about explanatory claims.

Perhaps most critically, the statistical framing demands that interpretability methods report not just point estimates but confidence sets or posterior distributions over explanations (in case of Bayesian framing). Just as we would not trust a clinical trial reporting effect sizes without confidence intervals, we may not trust interpretability claims without uncertainty quantification. When explanations are non-identifiable, this uncertainty will be large; when they are identifiable with finite data, uncertainty shrinks as observations accumulate.

5.2 Towards Useful and Identifiable Interpretability Tasks

Identifiability is not an intrinsic property of the model under study but of the interaction between $\mu$, $\mathcal{E}, D$, the model $f_{NN}$, and the behavior of interest $P_{\mathbf{U}}$. We might wonder what choices to make in order to improve the identifiability and usefulness of interpretability queries.

Query richness. The queries in the support of $\mu$ must be sufficiently discriminative to distinguish candidate explanations in $\mathcal{E}$. There is a fundamental trade-off between discriminative power and sample efficiency. If $\mu$ spreads probability mass over a large support, accurately estimating $L_\mu$ may require prohibitive amounts of interventional data. Conversely, concentrating $\mu$ on too few queries risks not singling out one explanation in $\mathcal{E}$.

Expressivity vs. parsimony in $\mathcal{E}$. Conversely, the hypothesis class must have sufficient capacity to approximate the queries well (low bias) but not so much flexibility that many distinct explanations all achieve low error (large equivalence classes, high variance, non-identifiability). This is akin to the classical bias-variance tradeoff, pointing toward standard fixes like regularization of the hypothesis class ([71]).

Human cognitive constraints. Interpretability is meant to facilitate human understanding. Empirical studies suggest people can mentally simulate models with only a handful of interacting components ([144, 145, 146, 147]). Explanations exceeding these structural limits may be technically correct yet fail to provide insight. Designing $\mathcal{E}$ with human simulability in mind ensures that understanding remains the end goal.

5.3 Opportunities for Future Work

Characterizing identifiability conditions.

A systematic theoretical program could characterize when specific $(\mu, \mathcal{E}, D)$ triplets are identifiable, mirroring similar efforts in causal inference ([134]) and unsupervised learning ([132, 133]). What symmetries and invariances are unavoidable in representation space, and when is identifiability up to such equivalences acceptable? Constructing a taxonomy of identifiable interpretability tasks would provide actionable guidance for practical scenarios.

Bayesian interpretability and uncertainty quantification.

Bayesian approaches offer an elegant framework for handling non-identifiability and quantifying uncertainty ([148]). Specifically, one could specify a prior distribution $\pi(e)$ over the explanation class $\mathcal{E}$, encoding structural preferences (e.g., sparsity, modularity) or incorporating prior information from related studies. Then, the likelihood model $P(\mathcal{T}_n \mid e)$ describes how computational traces are generated given explanation $e$. Finally, the posterior updates via Bayes' rule: $\pi(e \mid \mathcal{T}_n) \propto P(\mathcal{T}_n \mid e) \pi(e)$, refines beliefs as observations accumulate. Then, credible sets can quantify uncertainty. When explanations are non-identifiable, the posterior remains diffuse across an equivalence class; uncertainty quantification naturally reflects this fundamental ambiguity. Conversely, as more discriminative queries are observed, the posterior concentrates. This further provides a principled framework for active setup: strategically selecting queries from $\mu$ that maximally reduce posterior uncertainty.

Meta-analysis and cumulative science.

Meta-analytic methods ([149]) could coherently aggregate evidence across studies, accounting for heterogeneity in $\mu$, $\mathcal{E}$, and experimental conditions. Standardized effect size measures, pre-registration of analyses, and open sharing of collected computational traces would enable interpretability to become a cumulative science where knowledge systematically builds over time. In general, the solutions proposed by other fields ([150, 42]) discussed in the introduction now become available for interpretability.

Acknowledgments

Section Summary: The research was partly carried out at a French laboratory known as UMR 5217. It received funding support from the CNRS and from the MIAI institute at Grenoble-Alpes through two specific grants. The project also gained access to high-performance computing resources at IDRIS via an allocation provided by GENCI.

This work was partly conducted within the French research unit UMR 5217 and was supported by CNRS (grant ANR-22-CPJ2-0036-01) and by MIAI@Grenoble-Alpes (grant ANR-19-P3IA-0003). It was granted access to the HPC resources of IDRIS under the allocation 2025-AD011014834 made by GENCI.

Appendix

Section Summary: The appendix proposes using statistical hypothesis testing to validate results from neural network interpretability methods and avoid spurious “dead salmon” findings. It constructs null models by randomizing weights or labels while preserving architecture, then compares an observed explanation’s test statistic against those from many randomized runs to compute a p-value that indicates whether the result exceeds what random computation would produce. Experiments on probing tasks for sentiment, part-of-speech tags, and geospatial knowledge in models such as BERT and Pythia illustrate that many apparently strong effects vanish under this stricter test, while a few robust signals remain.

A. Fixing Dead Salmons with Hypothesis Testing

Consider an interpretability method $M$ that aims to explain a neural network $f$ for some input behavior $P_{\mathbf{U}}$, we note $\mathfrak{C}$ the tuple $(f, P_{\mathbf{U}})$ as done in the main paper. The method produces an explanation $\hat{e}$ from finite observations from $\mathfrak{C}$, possibly under interventions. This tentative explanation could take the form of a circuit, a set of important features, concept activation vectors, or any other hypothesis class. We might wonder how to prevent dead salmon artifacts from arising with the interpretability method $M$?

A simple, direct, and natural solution is to frame this question as a hypothesis test against a null hypothesis where the observed explanation arises from random computation. Already, for probing, [11] discusses the possibility of comparing the probe against a probe trained on random embeddings. The general idea is to construct a family of null models represented by a distribution $P_{\tilde{\mathfrak{C}}}$, which preserves the network’s architectural properties while disrupting the specific computational mechanisms we aim to explain. Such null models can be obtained, for example, via full weight randomization, random orthogonal transformations of representations, or label shuffling (recovering the standard permutation test).

For a given interpretability method, we define a test statistic $T(\hat{e}, \mathfrak{C})$ that quantifies explanatory fit for the interpretability task at hand. For example, for probing methods, $T$ could be test accuracy; for circuit discovery, $T$ could measure behavioral fidelity; for attribution methods, $T$ could quantify the correlation between attribution scores and actual intervention effects.

Applying the intepretability method $M$ to one null model $\tilde{\mathfrak{C}}^{(b)}$ from the randomized family yields explanations $\tilde{e}^{(b)} = M(\tilde{\mathfrak{C}}^{(b)})$ and corresponding null statistics $T_{\text{null}}^{(b)} = T(\tilde{e}^{(b)}, \tilde{\mathfrak{C}}^{(b)})$. Then, following standard procedure, the Monte Carlo estimated $p$-value is:

$ \hat{p} = \frac{1 + \sum_{b=1}^B \mathbb{I}{T_{\text{null}}^{(b)} \ge T_{\text{obs}}}}{B + 1},\tag{4} $

where $T_{\text{obs}} = T(\hat{e}, \mathfrak{C})$. The addition of 1 to both the numerator and denominator ensures Type I error control: $\Pr(\hat{p} \le \alpha \mid H_0) \le \alpha, $ where $H_0$ is the null hypothesis ([151, 152]). By design, when the randomization includes full weight reinitialization, no dead salmon artifacts can remain.

A.1 Experiments

To illustrate the hypothesis test, we experiment with three probing tasks.

Sentiment Analysis (IMDb). We reuse the IMDB sentiment classification setup from Figure 1. For each layer of BERT-base-uncased, we extract the average sentence embedding and train a linear probe to predict binary sentiment. We also train probes on $k{=}20$ random reinitializations of the model, and evaluate statistical significance using the hypothesis test described above. All probes are trained and evaluated on 1000 sentences with 10-fold cross-validation. Figure 4(A) reports (i) the average probe accuracy at each layer for the pretrained model, the randomized models, and a random guessing baseline, and (ii) the corresponding effect sizes relative to random guessing and to randomized models. While all pretrained layers outperform random guessing with large effect sizes, none are statistically distinguishable from the random reinitializations under the new test. Later layers, however, show a clear upward trend in effect size relative to randomized models.

Syntactic Structure (POS Tagging). We next assess token-level syntactic information using POS-tagging probes ([153]). For each layer of BERT-base-uncased, we extract contextual token embeddings and train logistic regression probes on a subset of CoNLL-2003, one probe per layer that should work for all tokens and all POS tags. As above, we also train probes on $k{=}20$ random reinitializations and apply the same statistical test. Probes are evaluated with 10-fold cross-validation on 500 sentences. Figure 4(B) reports the layer-wise probe accuracy and effect sizes relative to a majority baseline and to randomized models. Consistent with prior work, POS accuracy peaks in middle layers ([153]). However, when tested against randomized models rather than random guessing, only the middle layers remain statistically above chance, and the effect sizes are substantially reduced. This shows that testing against random computations eliminates many positive findings while still allowing for genuine positive discoveries where structure is robust.

**Figure 4:** (A) Sentiment analysis experiment where probes on pretrained BERT are compared against probes trained on random computation. (B) Same experiment based on predicting syntactic labels (POS tags). (C) Reproducing the first experiment of Table 2 in [154], probing for indications of world models on pythia-160m.

World Models (Space and Time). Finally, we investigate the emergence of linear representations of space using the "world places" dataset from ([154]). Using pythia-160m, we extract average residual stream activations on each token of place names and train linear ridge regression probes to predict their geospatial coordinates (latitude and longitude). We compare the pretrained model against $k=20$ baselines where transformer block weights are randomized while embeddings remain fixed. Probes are evaluated using $R^2$ scores with 10-fold cross-validation. Figure 4(C) reveals that raw embeddings (Layer 0) contain latent spatial structure ($R^2 \approx 0.12$), significantly outperforming random guessing ($Z \approx 100$). Passing these embeddings through randomized transformer blocks decreases linear readout ($R^2 \approx 0.38$). In contrast, the pretrained model's layers slightly improve this spatial linearity relative to the random baseline, suggesting that deeper layers progressively construct a more coherent spatial representation. By the final layers, the learned structure statistically surpasses the random baseline ($Z \approx 25$), confirming that the model eventually learns to encode space explicitly beyond the geometry inherent in the embeddings.

::: {caption="Table 1: Interpretability methods as instances of the statistical–causal framework of surrogate models. Each method specifies a hypothesis class E, causal query family q(mathfrakC), and associated error measure D quantifying how faithfully the surrogate answers the queries."}

:::

References

Section Summary: This section compiles academic papers and articles that examine problems with reliability and interpretation in brain imaging research, such as inflated false positives from small samples or flawed statistics. It also includes works on making artificial intelligence systems more understandable, covering topics like detecting biases, explaining model decisions, and assessing whether advanced techniques truly reveal how neural networks process information. Many entries highlight parallels between challenges in neuroscience and the growing field of AI interpretability.

[1] Bennett et al. (2009). Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: an argument for multiple comparisons correction. Neuroimage. 47(Suppl 1). pp. S125.

[2] Botvinik-Nezer et al. (2020). Variability in the analysis of a single neuroimaging dataset by many teams. Nature. 582(7810). pp. 84–88.

[3] Eklund et al. (2016). Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates. Proceedings of the national academy of sciences. 113(28). pp. 7900–7905.

[4] Vul et al. (2009). Voodoo correlations in social neuroscience. Perspectives on psychological Science. 4(3). pp. 274–290.

[5] Button et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature reviews neuroscience. 14(5). pp. 365–376.

[6] Marek et al. (2022). Reproducible brain-wide association studies require thousands of individuals. Nature. 603(7902). pp. 654–660.

[7] Lyon, Louisa (2017). Dead salmon and voodoo correlations: should we be sceptical about functional MRI?. Brain. 140(8). pp. e53–e53.

[8] Poldrack, Russell A (2006). Can cognitive processes be inferred from neuroimaging data?. Trends in cognitive sciences. 10(2). pp. 59–63.

[9] Duncan, John and Owen, Adrian M (2000). Common regions of the human frontal lobe recruited by diverse cognitive demands. Trends in neurosciences. 23(10). pp. 475–483.

[10] Adebayo et al. (2018). Sanity checks for saliency maps. Advances in neural information processing systems. 31.

[11] Ravichander et al. (2021). Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. pp. 3363–3377. doi:10.18653/v1/2021.eacl-main.295. https://aclanthology.org/2021.eacl-main.295.

[12] Thomas Heap et al. (2025). Sparse Autoencoders Can Interpret Randomly Initialized Transformers. https://arxiv.org/abs/2501.17727. arXiv:2501.17727.

[13] Méloux et al. (2025). Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?. In The Thirteenth International Conference on Learning Representations.

[14] Denis Sutter et al. (2025). The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?. https://arxiv.org/abs/2507.08802. arXiv:2507.08802.

[15] Mehrabi et al. (2021). A survey on bias and fairness in machine learning. ACM computing surveys (CSUR). 54(6). pp. 1–35.

[16] Barnes, Emily and Hutson, James (2024). Navigating the complexities of ai: The critical role of interpretability and explainability in ensuring transparency and trust. International Journal of Multidisciplinary and Current Educational Research. 6(3).

[17] Dhanesh Ramachandram et al. (2025). Transparent AI: The Case for Interpretability and Explainability. https://arxiv.org/abs/2507.23535. arXiv:2507.23535.

[18] Kim, Jinkyu and Canny, John (2017). Interpretable learning for self-driving cars by visualizing causal attention. In Proceedings of the IEEE international conference on computer vision. pp. 2942–2950.

[19] Zech et al. (2018). Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine. 15(11). pp. e1002683.

[20] Caruana et al. (2015). Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. pp. 1721–1730.

[21] Kevin Meng et al. (2022). Locating and Editing Factual Associations in GPT. Advances in Neural Information Processing Systems. 36.

[22] Giovanni Monea et al. (2024). A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia. https://arxiv.org/abs/2312.02073. arXiv:2312.02073.

[23] Nguyen et al. (2025). Deploying Interpretability to Production with Rakuten: SAE Probes for PII Detection. Goodfire Research.

[24] Alejandro Barredo Arrieta et al. (2019). Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI. https://arxiv.org/abs/1910.10045. arXiv:1910.10045.

[25] Kristofik, Andrej (2025). Bias in AI (Supported) Decision Making: Old Problems, New Technologies. In International Journal for Court Administration. pp. 1.

[26] Lepori et al. (2025). Racing Thoughts: Explaining Contextualization Errors in Large Language Models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). pp. 3020–3036. doi:10.18653/v1/2025.naacl-long.155. https://aclanthology.org/2025.naacl-long.155/.

[27] Barrett et al. (2019). Analyzing biological and artificial neural networks: challenges with opportunities for synergy?. Current opinion in neurobiology. 55. pp. 55–64.

[28] Lindsey et al. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread. https://transformer-circuits.pub/2025/attribution-graphs/biology.html.

[29] Allen-Zhu, Zeyuan and Li, Yuanzhi (2023). Physics of Language Models: Part 1, Learning Hierarchical Language Structures. SSRN Electronic Journal.

[30] Allen-Zhu, Zeyuan (2024). ICML 2024 Tutorial: Physics of Language Models. Project page: https://physics.allen-zhu.com/.

[31] Ghorbani et al. (2019). Interpretation of Neural Networks Is Fragile. Proceedings of the AAAI Conference on Artificial Intelligence. 33(01). pp. 3681-3688. doi:10.1609/aaai.v33i01.33013681. https://ojs.aaai.org/index.php/AAAI/article/view/4252.

[32] Kindermans et al. (2019). The (Un)reliability of Saliency Methods. doi:10.1007/978-3-030-28954-$6_14. $https://doi.org/10.1007/978-3-030-28954-$6_14$.

[33] Zhang et al. (2025). Saliency Maps Give a False Sense of Explanability to Image Classifiers: An Empirical Evaluation across Methods and Metrics. In Proceedings of the 16th Asian Conference on Machine Learning. pp. 479–494. https://proceedings.mlr.press/v260/zhang25a.html.

[34] Zafar et al. (2021). On the Lack of Robust Interpretability of Neural Text Classifiers. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pp. 3730–3740. doi:10.18653/v1/2021.findings-acl.327. https://aclanthology.org/2021.findings-acl.327.

[35] Jason Hoelscher-Obermaier et al. (2023). Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark. https://arxiv.org/abs/2305.17553. arXiv:2305.17553.

[36] Dombrowski et al. (2019). Explanations can be manipulated and geometry is to blame. In Advances in Neural Information Processing Systems. pp. . https://proceedings.neurips.cc/paper_files/paper/2019/file/bb836c01cdc9120a9c984c525e4b1a4a-Paper.pdf.

[37] Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science. 349(6251). pp. aac4716. doi:10.1126/science.aac4716. https://www.science.org/doi/abs/10.1126/science.aac4716.

[38] Simmons et al. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science. 22(11). pp. 1359–1366.

[39] Ioannidis, John PA (2005). Why most published research findings are false. PLoS medicine. 2(8). pp. e124.

[40] Schimmack, Ulrich (2020). A meta-psychological perspective on the decade of replication failures in social psychology.. Canadian Psychology/Psychologie Canadienne. 61(4). pp. 364.

[41] Munafò et al. (2017). A manifesto for reproducible science. Nature human behaviour. 1(1). pp. 0021.

[42] Korbmacher et al. (2023). The replication crisis has led to positive structural, procedural, and community changes. Communications Psychology. 1(1). pp. 3.

[43] Pearl, Judea (2009). Causality: Models, Reasoning and Inference. Cambridge University Press.

[44] Imbens, Guido W and Rubin, Donald B (2015). Causal inference in statistics, social, and biomedical sciences. Cambridge university press.

[45] Angrist, Joshua D and Pischke, Jörn-Steffen (2009). Mostly harmless econometrics: An empiricist's companion. Princeton university press.

[46] Heckman, James J (2007). The economics, technology, and neuroscience of human capability formation. Proceedings of the national Academy of Sciences. 104(33). pp. 13250–13255.

[47] Miller, Tim (2019). Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence. 267. pp. 1–38.

[48] Iwan Williams et al. (2025). Mechanistic Interpretability Needs Philosophy. https://arxiv.org/abs/2506.18852. arXiv:2506.18852.

[49] Timothy P. Lillicrap and Konrad P. Kording (2019). What does it mean to understand a neural network?. https://arxiv.org/abs/1907.06374. arXiv:1907.06374.

[50] Lipton, Zachary C (2018). The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.. Queue. 16(3). pp. 31–57.

[51] Psillos, Stathis (2005). Scientific realism: How science tracks truth. Routledge.

[52] Chakravartty, Anjan (2011). Scientific realism. The Stanford Encyclopedia of Philosophy. (Summer 2017 Edition). https://plato.stanford.edu/archives/sum2017/entries/scientific-realism.

[53] Rudin, Cynthia (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence. 1(5). pp. 206–215.

[54] Páez, Andrés (2019). The Pragmatic Turn in Explainable Artificial Intelligence (XAI). Minds and Machines. 29(3). pp. 441–459. doi:10.1007/s11023-019-09502-w. http://dx.doi.org/10.1007/s11023-019-09502-w.

[55] Saphra, Naomi and Wiegreffe, Sarah (2024). Mechanistic?. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP. pp. 480–498.

[56] Andy Zou et al. (2025). Representation Engineering: A Top-Down Approach to AI Transparency. https://arxiv.org/abs/2310.01405. arXiv:2310.01405.

[57] John Dewey (1948). Reconstruction in Philosophy. Dover Publications.

[58] Chang, Hasok (2004). Inventing temperature: Measurement and scientific progress. Oxford University Press.

[59] Potochnik, Angela (2017). Idealization and the Aims of Science.

[60] Van Fraassen, Bas C (1980). The scientific image. Oxford University Press.

[61] Cartwright, Nancy (1983). How the laws of physics lie. Oxford University Press.

[62] Karen Simonyan et al. (2014). Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. https://arxiv.org/abs/1312.6034. arXiv:1312.6034.

[63] Sundararajan et al. (2017). Axiomatic attribution for deep networks. In International conference on machine learning. pp. 3319–3328.

[64] Bilodeau et al. (2024). Impossibility theorems for feature attribution. Proceedings of the National Academy of Sciences. 121(2). pp. e2304406120.

[65] Conneau et al. (2018). What you can cram into a single &!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 2126–2136. doi:10.18653/v1/P18-1198. https://aclanthology.org/P18-1198.

[66] Hewitt, John and Manning, Christopher D. (2019). A Structural Probe for Finding Syntax in Word Representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 4129–4138. doi:10.18653/v1/N19-1419. https://aclanthology.org/N19-1419.

[67] Hall Maudslay, Rowan and Cotterell, Ryan (2021). Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 124–131. doi:10.18653/v1/2021.naacl-main.11. https://aclanthology.org/2021.naacl-main.11.

[68] Voita, Elena and Titov, Ivan (2020). Information-Theoretic Probing with Minimum Description Length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 183–196. doi:10.18653/v1/2020.emnlp-main.14. https://aclanthology.org/2020.emnlp-main.14.

[69] Zhu, Zining and Rudzicz, Frank (2020). An information theoretic view on selecting linguistic probes. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 9251–9262. doi:10.18653/v1/2020.emnlp-main.744. https://aclanthology.org/2020.emnlp-main.744.

[70] Pimentel et al. (2020). Information-Theoretic Probing for Linguistic Structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 4609–4622. doi:10.18653/v1/2020.acl-main.420. https://aclanthology.org/2020.acl-main.420.

[71] Belinkov, Yonatan (2022). Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics. 48(1). pp. 207–219. doi:10.1162/coli_a_00422. https://aclanthology.org/2022.cl-1.7.

[72] Elazar et al. (2021). Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals. Transactions of the Association for Computational Linguistics. 9. pp. 160–175. doi:10.1162/tacl_a_00359. https://aclanthology.org/2021.tacl-1.10.

[73] Hoagy Cunningham et al. (2023). Sparse Autoencoders Find Highly Interpretable Features in Language Models. https://arxiv.org/abs/2309.08600. arXiv:2309.08600.

[74] Yun et al. (2021). Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors. In Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures. pp. 1–10. doi:10.18653/v1/2021.deelio-1.1. https://aclanthology.org/2021.deelio-1.1.

[75] Bricken et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread.

[76] Templeton et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread. https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html.

[77] Lovis Heindrich et al. (2025). Do Sparse Autoencoders Generalize? A Case Study of Answerability. https://arxiv.org/abs/2502.19964. arXiv:2502.19964.

[78] Subhash Kantamneni et al. (2025). Are Sparse Autoencoders Useful? A Case Study in Sparse Probing. https://arxiv.org/abs/2502.16681. arXiv:2502.16681.

[79] Aaron J. Li et al. (2025). Interpretability Illusions with Sparse Autoencoders: Evaluating Robustness of Concept Representations. https://arxiv.org/abs/2505.16004. arXiv:2505.16004.

[80] Kim et al. (2018). Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning. pp. 2668–2677.

[81] Bau et al. (2017). Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6541–6549.

[82] Sanchit Sinha and Aidong Zhang (2025). A Comprehensive Survey on the Risks and Limitations of Concept-based Models. https://arxiv.org/abs/2506.04237. arXiv:2506.04237.

[83] Halil Ibrahim Aysel et al. (2025). Concept-Based Explainable Artificial Intelligence: Metrics and Benchmarks. https://arxiv.org/abs/2501.19271. arXiv:2501.19271.

[84] Tolga Bolukbasi et al. (2021). An Interpretability Illusion for BERT. https://arxiv.org/abs/2104.07143. arXiv:2104.07143.

[85] Angus Nicolson et al. (2025). Explaining Explainability: Recommendations for Effective Use of Concept Activation Vectors. https://arxiv.org/abs/2404.03713. arXiv:2404.03713.

[86] Ramaswamy et al. (2023). Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10932–10941.

[87] Vihari Piratla et al. (2024). Estimation of Concept Explanations Should be Uncertainty Aware. https://arxiv.org/abs/2312.08063. arXiv:2312.08063.

[88] Pearl, Judea (2012). The causal mediation formula—a guide to the assessment of pathways and mechanisms. Prevention science. 13. pp. 426–436.

[89] Vig et al. (2020). Investigating Gender Bias in Language Models Using Causal Mediation Analysis. In Advances in Neural Information Processing Systems. pp. 12388–12401. https://proceedings.neurips.cc/paper_files/paper/2020/file/92650b2e92217715fe312e6fa7b90d82-Paper.pdf.

[90] Finlayson et al. (2021). Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 1828–1843. doi:10.18653/v1/2021.acl-long.144. https://aclanthology.org/2021.acl-long.144.

[91] Syed et al. (2023). Attribution patching outperforms automated circuit discovery. arXiv preprint arXiv:2310.10348.

[92] Aaron Mueller et al. (2024). The Quest for the Right Mediator: A History, Survey, and Theoretical Grounding of Causal Interpretability. https://arxiv.org/abs/2408.01416. arXiv:2408.01416.

[93] Marc Canby et al. (2025). How Reliable are Causal Probing Interventions?. https://arxiv.org/abs/2408.15510. arXiv:2408.15510.

[94] Fred Zhang and Neel Nanda (2024). Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. https://arxiv.org/abs/2309.16042. arXiv:2309.16042.

[95] Thomas McGrath et al. (2023). The Hydra Effect: Emergent Self-repair in Language Model Computations. https://arxiv.org/abs/2307.15771. arXiv:2307.15771.

[96] Schaffer, Jonathan (2003). Overdetermining causes. Philosophical Studies: An International Journal for Philosophy in the Analytic Tradition. 114(1/2). pp. 23–45.

[97] Sider, Theodore (2003). What's so bad about overdetermination?.

[98] Dyrkolbotn, Sjur K (2017). On Preemption and Overdetermination in Formal Theories of Causality. Electronic Proceedings in Theoretical Computer Science. 259. pp. 1–15. doi:10.4204/eptcs.259.1. http://dx.doi.org/10.4204/EPTCS.259.1.

[99] Olah et al. (2020). Zoom In: An Introduction to Circuits. Distill. doi:10.23915/distill.00024.001.

[100] Jacob Dunefsky et al. (2024). Transcoders Find Interpretable LLM Feature Circuits. https://arxiv.org/abs/2406.11944. arXiv:2406.11944.

[101] Adam Davies and Ashkan Khakzar (2024). The Cognitive Revolution in Interpretability: From Explaining Behavior to Interpreting Representations and Algorithms. https://arxiv.org/abs/2408.05859. arXiv:2408.05859.

[102] Conmy et al. (2023). Towards Automated Circuit Discovery for Mechanistic Interpretability. In Advances in Neural Information Processing Systems. pp. 16318–16352. https://proceedings.neurips.cc/paper_files/paper/2023/file/34e1dbe95d34d7ebaf99b9bcaeb5b2be-Paper-Conference.pdf.

[103] Geiger et al. (2022). Faithful, Interpretable Model Explanations via Causal Abstraction. https://ai.stanford.edu/blog/causal-abstraction/.

[104] Geiger et al. (2022). Inducing Causal Structure for Interpretable Neural Networks. In Proceedings of the 39th International Conference on Machine Learning. pp. 7324–7338. https://proceedings.mlr.press/v162/geiger22a.html.

[105] Beckers, Sander and Halpern, Joseph Y. (2019). Abstracting Causal Models. Proceedings of the AAAI Conference on Artificial Intelligence. 33(01). pp. 2678-2685. doi:10.1609/aaai.v33i01.33012678. https://ojs.aaai.org/index.php/AAAI/article/view/4117.

[106] Lee Sharkey et al. (2025). Open Problems in Mechanistic Interpretability. https://arxiv.org/abs/2501.16496. arXiv:2501.16496.

[107] Aleksandar Makelov et al. (2023). Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching. https://arxiv.org/abs/2311.17030. arXiv:2311.17030.

[108] Kevin Wang et al. (2022). Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. https://arxiv.org/abs/2211.00593. arXiv:2211.00593.

[109] Victoria R. Li et al. (2025). Can Interpretation Predict Behavior on Unseen Data?. https://arxiv.org/abs/2507.06445. arXiv:2507.06445.

[110] Maxime Méloux et al. (2025). Mechanistic Interpretability as Statistical Estimation: A Variance Analysis of EAP-IG. https://arxiv.org/abs/2510.00845. arXiv:2510.00845.

[111] Marasovic et al. (2022). Few-Shot Self-Rationalization with Natural Language Prompts. In Findings of the Association for Computational Linguistics: NAACL 2022. pp. 410–424. https://aclanthology.org/2022.findings-naacl.31.

[112] Wiegreffe et al. (2022). Reframing Human-AI Collaboration for Generating Free-Text Explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 632–658. https://aclanthology.org/2022.naacl-main.47.

[113] Rohan Ajwani et al. (2024). LLM-Generated Black-box Explanations Can Be Adversarially Helpful. https://arxiv.org/abs/2405.06800. arXiv:2405.06800.

[114] Tamera Lanham et al. (2023). Measuring Faithfulness in Chain-of-Thought Reasoning. https://arxiv.org/abs/2307.13702. arXiv:2307.13702.

[115] Iván Arcuschin et al. (2025). Chain-of-Thought Reasoning in the Wild is not Always Faithful. In Workshop on Reasoning and Planning for Large Language Models. https://openreview.net/forum?id=L8094Whth0.

[116] Miles Turpin et al. (2023). Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. https://arxiv.org/abs/2305.04388. arXiv:2305.04388.

[117] Jacovi et al. (2021). Contrastive explanations for model interpretability. arXiv preprint arXiv:2103.01378.

[118] Rogers et al. (2021). A primer in BERTology: What we know about how BERT works. Transactions of the association for computational linguistics. 8. pp. 842–866.

[119] Hagendorff et al. (2023). Machine psychology. arXiv preprint arXiv:2303.13988.

[120] Kosslyn, Stephen M (1999). If neuroimaging is the answer, what is the question?. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences. 354(1387). pp. 1283–1294.

[121] Logothetis, Nikos K (2008). What we can do and what we cannot do with fMRI. Nature. 453(7197). pp. 869–878.

[122] Churchland, Patricia S and Sejnowski, Terrence J (1988). Perspectives on cognitive neuroscience. Science. 242(4879). pp. 741–745.

[123] Teney et al. (2022). Predicting Is Not Understanding: Recognizing And Addressing Underspecification In Machine Learning. In ECCV 2022: 17th European Conference on Computer Vision. pp. 458–476. doi:10.1007/978-3-031-20050-2_27. https://doi.org/10.1007/978-3-031-20050-2_27.

[124] D'Amour et al. (2022). Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research. 23(1).

[125] Frankle, Jonathan and Carbin, Michael (2019). The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks.. In ICLR. http://dblp.uni-trier.de/db/conf/iclr/iclr2019.html#FrankleC19.

[126] G Tononi et al. (1994). A measure for brain complexity: relating functional segregation and integration in the nervous system.. Proceedings of the National Academy of Sciences. 91(11). pp. 5033-5037. doi:10.1073/pnas.91.11.5033. https://www.pnas.org/doi/abs/10.1073/pnas.91.11.5033.

[127] Loosemore, Richard PW (2012). The Complex Cognitive Systems Manifesto.

[128] Advait Sarkar (2022). Is explainable AI a race against model complexity?. https://arxiv.org/abs/2205.10119. arXiv:2205.10119.

[129] Lindsay, Grace W and Bau, David (2023). Testing methods of neural systems understanding. Cognitive Systems Research. 82. pp. 101156.

[130] Casella, George and Berger, Roger (2024). Statistical inference. CRC press.

[131] Elizabeth S. Allman et al. (2009). Identifiability of parameters in latent structure models with many observed variables. Annals of Statistics. 37(6A). pp. 3099–3132.

[132] Francesco Locatello et al. (2019). Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations. In Proceedings of the 36th International Conference on Machine Learning (ICML). pp. 4114–4124.

[133] Ilyes Khemakhem et al. (2020). Variational Autoencoders and Nonlinear ICA: A Unifying Framework. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS). pp. 2207–2217.

[134] Ilya Shpitser and Judea Pearl (2008). Complete Identification Methods for the Causal Hierarchy. Journal of Machine Learning Research. 9. pp. 1941–1979.

[135] Claudia Shi et al. (2024). Hypothesis Testing the Circuit Hypothesis in LLMs. https://arxiv.org/abs/2410.13032. arXiv:2410.13032.

[136] Cox, David Roxbee (2006). Principles of statistical inference. Cambridge university press.

[137] Lehmann, Erich Leo and Casella, George (1998). Theory of point estimation. Springer.

[138] Olah et al. (2018). The Building Blocks of Interpretability. Distill. doi:10.23915/distill.00010.

[139] Cammarata et al. (2020). Thread: Circuits. Distill. doi:10.23915/distill.00024.

[140] Atticus Geiger et al. (2025). Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability. https://arxiv.org/abs/2301.04709. arXiv:2301.04709.

[141] Senetaire et al. (2023). Explainability as statistical inference. In Proceedings of the 40th International Conference on Machine Learning. pp. 30584–30612. https://proceedings.mlr.press/v202/senetaire23a.html.

[142] Jesse Vig et al. (2020). Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias. https://arxiv.org/abs/2004.12265. arXiv:2004.12265.

[143] James F. Woodward (2003). Making Things Happen: A Theory of Causal Explanation. Oxford University Press.

[144] Lombrozo, Tania (2006). The structure and function of explanations. Trends in cognitive sciences. 10(10). pp. 464–470.

[145] Wilkenfeld, Daniel A (2013). Understanding as representation manipulability. Synthese. 190(6). pp. 997–1016.

[146] Keil, Frank C (2006). Explanation and understanding. Annual Reviews of Psychology. 57(1). pp. 227–254.

[147] Hassija et al. (2024). Interpreting black-box models: a review on explainable artificial intelligence. Cognitive Computation. 16(1). pp. 45–74.

[148] Gelman et al. (2013). Bayesian Data Analysis. CRC. https://stat.columbia.edu/~gelman/book/.

[149] Borenstein et al. (2021). Introduction to meta-analysis. John wiley & sons.

[150] Poldrack et al. (2017). Scanning the horizon: towards transparent and reproducible neuroimaging research. Nature reviews neuroscience. 18(2). pp. 115–126.

[151] North et al. (2002). A note on the calculation of empirical P values from Monte Carlo procedures. The American Journal of Human Genetics. 71(2). pp. 439–441.

[152] Phipson, Belinda and Smyth, Gordon K (2010). Permutation P-values should never be zero: calculating exact P-values when permutations are randomly drawn. Statistical applications in genetics and molecular biology. 9. pp. Article39.

[153] Tenney et al. (2019). BERT Rediscovers the Classical NLP Pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 4593–4601. doi:10.18653/v1/P19-1452. https://aclanthology.org/P19-1452.

[154] Wes Gurnee and Max Tegmark (2024). Language Models Represent Space and Time. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=jE8xbmvFin.