Attention is not Explanation

Sarthak JainByron C. Wallace

article2019NAACL2,016 citations

Demonstrates that attention weights in neural language models fail to provide reliable explanations for predictions, proving through empirical tests that attention rarely correlates with gradient-based feature importance and that distinct attention distributions can yield identical outputs.

Listen

Modern language models widely incorporate attention mechanisms, which assign dynamic numerical weights to different words in a text to improve task accuracy. Because these weights highlight specific words, researchers and practitioners routinely display them as visual heatmaps and treat them as faithful explanations of why a model reached a particular conclusion. This practice creates a serious operational risk: decision-makers may assume an artificial intelligence system is transparent and accountable when its internal reasoning is actually unverified.

The article set out to rigorously test whether attention weights provide meaningful, faithful explanations for neural network predictions. Specifically, the authors evaluated whether attention weights agree with established measures of word importance and whether changing these weights produces predictable shifts in final predictions.

To test this, the authors conducted comprehensive experiments across standard text classification, question answering, and natural language inference tasks spanning several domains, including movie reviews, news articles, and medical records. They analyzed models that use recurrent neural networks to build contextual representations, comparing them against simpler non-recurrent baselines. The methodology focused on two central tests: calculating statistical rank correlation between attention weights and direct feature importance metrics (such as word-erasure and gradient-based impact), and systematically generating counterfactual or "adversarial" attention distributions to see if very different word weightings would change the final output.

The investigation produced three critical findings. First, for standard recurrent models, attention weights show only weak and inconsistent correlation with established feature importance metrics, with average correlation coefficients generally hovering at or below 0.50. Second, randomly shuffling attention weights across words frequently causes little to no change in the model's final prediction, often resulting in median output shifts of less than 0.01. Third, the authors successfully constructed adversarial attention distributions that attended to entirely different words while keeping the output essentially identical to the original prediction. High agreement between attention and feature importance only appeared in non-recurrent, simple projection models, where correlation was roughly 0.375 points higher on average than in recurrent models.

These findings mean that highlighting words based on standard attention weights does not provide a reliable explanation of model behavior, especially when using complex, contextual text encoders. Relying on attention heatmaps to justify decisions introduces compliance, safety, and operational risks in high-stakes settings like clinical record analysis or legal document processing. Contrary to common belief, presenting a compelling visual heatmap creates an illusion of transparency that can easily mislead stakeholders.

Organizations should immediately stop treating raw attention weights as standalone explanations for high-stakes decision-making. Instead, technical teams needing explainability should adopt dedicated interpretability techniques, such as gradient-based attribution, leave-one-out feature testing, or specialized models that enforce hard, sparse selections of text. Further research and validation should focus on developing principled, structured attention mechanisms designed explicitly for interpretability.

These conclusions are supported with high confidence across text classification, question answering, and inference benchmarks. However, the evaluation focused on recurrent, convolutional, and feed-forward architectures on unstructured prediction tasks; it did not evaluate modern sequence-to-sequence translation architectures or bidirectional cross-attention mechanisms. While alternative feature importance metrics are not perfect ground truth, the consistent inability of attention weights to track model behavior confirms that practitioners must exercise caution when interpreting attention distributions.

Northeastern University attention mechanism word sense disambiguation distribution functions Attention Attention Weights Model

arXiv: 1902.10186 successar/AttentionExplanation

Paper: Sanity Checks for Saliency Maps, Julius Adebayo et al. (2018). This work establishes the foundational methodology of sanity-checking post-hoc saliency maps against model parameters and random baselines, which directly motivates and parallels the empirical debunking of attention weights as faithful explanations.
Paper: The Mythos of Model Interpretability, Zachary C. Lipton (2016). It provides the conceptual foundation and vocabulary separating mechanistic transparency from post-hoc explanation, clarifying the theoretical distinction tested empirically in the target paper.
Paper: Axiomatic Attribution for Deep Networks, Mukund Sundararajan et al. (2017). It introduces Integrated Gradients and the axiomatic foundation of gradient-based feature attribution, providing the core benchmark against which attention weights are compared for feature importance.
Paper: “Why Should I Trust You?”: Explaining the Predictions of Any Classifier, Marco Tulio Ribeiro et al. (2016). It defines local surrogate explanations and faithfulness criteria for neural predictions, setting the standard for interpretable explanations that attention is evaluated against.
Paper: Effective Approaches to Attention-based Neural Machine Translation, Minh-Thang Luong et al. (2015). It formalizes the standard global and local attention mechanisms for sequence modeling whose explanatory validity is scrutinized.
Paper: A Structured Self-attentive Sentence Embedding, Zhouhan Lin et al. (2017). It establishes structured self-attentive sentence embeddings widely used in NLP classification tasks evaluated in the study.
Paper: Hierarchical Attention Networks for Document Classification, Zichao Yang et al. (2016). It introduces hierarchical attention networks for text classification, representing a primary architecture often assumed to offer intuitive interpretability across words and sentences.
Paper: A Unified Approach to Interpreting Model Predictions, Scott M. Lundberg et al. (2017). It establishes the unified SHAP framework for game-theoretic feature attribution, representing a primary gold-standard attribution baseline compared with attention weights.

Paper: Synthesizer: Rethinking Self-Attention for Transformer Models, Yi Tay et al. (2021). Building on evidence that attention weights do not represent meaningful token-to-token explanatory relationships, this work develops Transformer models using synthetic and random attention weights that match standard performance.
Paper: The Dead Salmons of AI Interpretability, Maxime Méloux et al. (2025). It formalizes the theoretical reasons behind interpretability failures—such as those observed with attention weights—framing non-identifiability and underspecification as statistical and causal inference problems.
Paper: GNNExplainer: Generating Explanations for Graph Neural Networks, Rex Ying et al. (2019). It develops dedicated post-hoc optimization techniques for graph networks, demonstrating how purpose-built explainers overcome the limitations of raw attention and gradient weights.
Paper: Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI, Alejandro Barredo Arrieta et al. (2020). It incorporates the findings on the limits of attention transparency into a comprehensive taxonomy of deep learning explainability and responsible AI.

Abstract

Attention mechanisms have seen wide adoption in neural NLP models. In addition to improving predictive performance, these are often touted as affording transparency: models equipped with attention provide a distribution over attended-to input units, and this is often presented (at least implicitly) as communicating the relative importance of inputs. However, it is unclear what relationship exists between attention weights and model outputs. In this work, we perform extensive experiments across a variety of NLP tasks that aim to assess the degree to which attention weights provide meaningful `explanations' for predictions. We find that they largely do not. For example, learned attention weights are frequently uncorrelated with gradient-based measures of feature importance, and one can identify very different attention distributions that nonetheless yield equivalent predictions. Our findings show that standard attention modules do not provide meaningful explanations and should not be treated as though they do. Code for all experiments is available at this https URL.

Topics

attention mechanism word sense disambiguation distribution functions language model

Keywords

Attention Attention Weights Model Max Output BiLSTM arXiv Tion Explanation Adversarial

1 Introduction and Motivation
2 Preliminaries and Assumptions
3 Datasets and Tasks
4 \parExperiments
4.1 \parCorrelation Between Attention and Feature Importance Measures
4.2 \parCounterfactual Attention Weights
4.2.1 \parAttention Permutation
4.2.2 \parAdversarial Attention
5 \parRelated Work
6 \parDiscussion and Conclusions
7 \parAcknowledgements
References
A \parModel details
A.1 \parBiLSTM
A.2 \parCNN
A.3 \parAverage
B \parFurther details regarding attentional module of gradient
C \parGraphs
D \parAdversarial Heatmaps

Knowls

Knowl 1 — Adversarial Attention Optimization Formulation
model/method
To evaluate whether learned attention weights provide unique and faithful explanations of neural model predictions, an adversarial optimization framework searches for alternative attention distributions that maximally deviate from the model's induced attention weights while producing virtually identical output predictions.

Let $\hat{\alpha} = \{\hat{\alpha}_t\}_{t=1}^T$ be the observed attention distribution induced over an input sequence $x = (x_1, \dots, x_T)$ of length $T$ , where $\hat{\alpha} \in \Delta^{T-1}$ lies on the probability simplex. Let $\hat{y}(x, \alpha) \in \mathbb{R}^{|\mathcal{Y}|}$ denote the prediction distribution over a label set $\mathcal{Y}$ resulting from feeding the weighted context vector $h_\alpha = \sum_{t=1}^T \alpha_t h_t$ into the output decoder $\text{Dec}$ , keeping all underlying encoder hidden states $h = \text{Enc}(x)$ and model parameters $\theta$ fixed.

The objective seeks $k$ distinct adversarial attention distributions $\{\alpha^{(1)}, \dots, \alpha^{(k)}\}$ that maximize divergence from $\hat{\alpha}$ and pairwise divergence among each other, subject to an output variation bound $\epsilon$ :

$\max_{\alpha^{(1)}, \dots, \alpha^{(k)}} \sum_{i=1}^k \text{JSD}\left[\alpha^{(i)}, \hat{\alpha}\right] + \frac{1}{k(k - 1)} \sum_{i < j} \text{JSD}\left[\alpha^{(i)}, \alpha^{(j)}\right]$

$\text{subject to } \forall i \in \{1, \dots, k\}, \quad \text{TVD}\left[\hat{y}(x, \alpha^{(i)}), \hat{y}(x, \hat{\alpha})\right] \le \epsilon$

where $\text{JSD}$ is the Jensen-Shannon Divergence and $\text{TVD}$ is the Total Variation Distance. If an adversarial attention distribution $\alpha^{(i)}$ can be found with high $\text{JSD}$ from $\hat{\alpha}$ and $\text{TVD} \le \epsilon$ , the original attention weights $\hat{\alpha}$ do not provide a necessary or unique attribution for the model's decision.
Knowl 2 — Search Algorithm for Adversarial Attention Distributions
algorithm
The search for adversarial attention weights optimizes a continuous relaxation of the constrained divergence maximization problem using the Adam stochastic gradient optimizer.

The objective function incorporates a penalty parameter $\lambda$ to penalize prediction differences exceeding a tolerance threshold $\epsilon$ :

$\mathcal{L}\left(\{\alpha^{(i)}\}_{i=1}^k\right) = \sum_{i=1}^k \text{JSD}\left[\alpha^{(i)}, \hat{\alpha}\right] + \frac{1}{k(k-1)} \sum_{i < j} \text{JSD}\left[\alpha^{(i)}, \alpha^{(j)}\right] + \frac{\lambda}{k} \sum_{i=1}^k \max\left(0, \text{TVD}\left[\hat{y}(x, \alpha^{(i)}), \hat{y}(x, \hat{\alpha})\right] - \epsilon\right)$

In standard implementations, $\lambda = 500$ , while $\epsilon = 0.01$ for binary text classification and natural language inference, and $\epsilon = 0.05$ for question answering tasks. The primary outcome metric is the $\epsilon$ -maximal Jensen-Shannon Divergence ( $rans{\epsilon}\text{-max JSD}$ ), denoting the greatest distance from $\hat{\alpha}$ achieved by any candidate $\alpha^{(i)}$ satisfying the tolerance constraint.
Input: Input sequence $x$ , Query $Q$ , Encoder $\text{Enc}$ , Attention scorer $\phi$ , Decoder $\text{Dec}$ , Output threshold $\epsilon$ , Adversary count $k$ , Penalty weight $\lambda = 500$
Output: $\epsilon\text{-max JSD}$
$h \leftarrow \text{Enc}(x)$
$\hat{\alpha} \leftarrow \text{softmax}(\phi(h, Q))$
$\hat{y} \leftarrow \text{Dec}(h, \hat{\alpha})$
$\{\alpha^{(1)}, \dots, \alpha^{(k)}\} \leftarrow \text{Maximize relaxed objective via Adam SGD}$
for $i \leftarrow 1$ to $k$ do
     $\hat{y}^{(i)} \leftarrow \text{Dec}(h, \alpha^{(i)})$
     $\Delta \hat{y}^{(i)} \leftarrow \text{TVD}[\hat{y}, \hat{y}^{(i)}]$
     $\Delta \alpha^{(i)} \leftarrow \text{JSD}[\hat{\alpha}, \alpha^{(i)}]$
end for
$\epsilon\text{-max JSD} \leftarrow \max_{i} \mathbb{I}[\Delta \hat{y}^{(i)} \le \epsilon] \cdot \Delta \alpha^{(i)}$
return $\epsilon\text{-max JSD}$
Knowl 3 — Empirical Invariance of Predictions Under Adversarial Attention Configurations
empirical result
Across text classification, Question Answering (QA), and Natural Language Inference (NLI) benchmarks using Bidirectional Recurrent Neural Network (BiRNN/BiLSTM) and Convolutional Neural Network (CNN) encoders, adversarial attention distributions routinely achieve $\epsilon$ -max Jensen-Shannon Divergence values near the theoretical ceiling of $\ln(2) \approx 0.693$ while maintaining output total variation distance within $\epsilon = 0.01$ (or $\epsilon = 0.05$ for QA).

Key empirical properties identified:
1. Prediction Invariance to Divergent Attention: For the vast majority of instances in datasets including SST, IMDB, 20 Newsgroups, AG News, CNN-QA, bAbI, and SNLI, alternative attention distributions can be constructed that place weight on entirely different tokens while producing identical model outputs.
2. Weak Buffering from Peaky Attention: Even when original attention weights are highly concentrated on specific features (e.g., maximum attention weight $\max(\hat{\alpha}) \in [0.75, 1.0]$ ), adversarial attention distributions yielding near-identical outputs remain easily discoverable, exhibiting only a very weak negative correlation with max attention.
3. Domain Exception in High-Precision Signals: In specialized medical document classification tasks (e.g., MIMIC ICD-9 Diabetes identification), positive instances containing explicit, sparse clinical trigger keywords exhibit lower adversarial JSD, indicating that attention perturbations in the presence of strong localized evidence do induce output changes; however, this behavior is an exception rather than the general norm across NLP tasks.
Knowl 4 — Disconnected-Graph Gradient and Leave-One-Out Feature Importance Estimation
algorithm
To evaluate whether attention weights agree with established measures of feature importance, token-level importances are quantified using first-order input gradients and leave-one-out (LOO) representation erasures, and subsequently compared to attention weights via Kendall's rank correlation coefficient $\tau$ .

In the gradient calculation, the computation graph is intentionally disconnected at the attention layer $\hat{\alpha}$ . This ensures that gradients do not backpropagate through the attention mechanism itself, treating the attention distribution as a fixed weighting over input positions and isolating the sensitivity of the output to infinitesimal perturbations of individual word vectors under that fixed attention.
Input: Input sequence $x$ with $T$ one-hot token vectors $x_t \in \mathbb{R}^{|V|}$ , Query vector $Q$ , Encoder $\text{Enc}$ , Attention function $\phi$ , Dense decoder $\text{Dec}$
Output: Kendall's rank correlations $\tau_g$ (gradient) and $\tau_{\text{loo}}$ (leave-one-out)
$h \leftarrow \text{Enc}(x)$
$\hat{\alpha} \leftarrow \text{softmax}(\phi(h, Q))$
$\hat{y} \leftarrow \text{Dec}(h, \hat{\alpha})$
for $t \leftarrow 1$ to $T$ do
$g_t \leftarrow \left| \sum_{w=1}^{|V|} \mathbb{I}[x_{tw} = 1] \frac{\partial y}{\partial x_{tw}} \right|$
$\Delta \hat{y}_t \leftarrow \text{TVD}(\hat{y}(x_{-t}), \hat{y}(x))$
end for
$\tau_g \leftarrow \text{Kendall-}\tau(\hat{\alpha}, g)$
$\tau_{\text{loo}} \leftarrow \text{Kendall-}\tau(\hat{\alpha}, \Delta \hat{y})$
return $\tau_g, \tau_{\text{loo}}$
Here $x_{-t}$ denotes the input sequence with the token at index $t$ deleted, and $\text{TVD}$ denotes the Total Variation Distance.

Knowl 5 — Correlation of Attention Weights with Gradient and Leave-One-Out Importance Measures

data/table

Kendall's rank correlation $\tau$ measures the agreement between learned attention weights $\hat{\alpha}$ and two input feature importance baselines: gradient-based importance ( $\tau_g$ ) and leave-one-out output difference ( $\tau_{\text{loo}}$ ).

		Gradient (BiLSTM) $\tau_g$		Gradient (Average) $\tau_g$		Leave-One-Out (BiLSTM) $\tau_{\text{loo}}$
Dataset	Class	Mean $\pm$ Std.	Sig. Frac.	Mean $\pm$ Std.	Sig. Frac.	Mean $\pm$ Std.	Sig. Frac.
SST	0	0.34 $\pm$ 0.21	0.48	0.61 $\pm$ 0.20	0.87	0.27 $\pm$ 0.19	0.33
	1	0.36 $\pm$ 0.21	0.49	0.60 $\pm$ 0.21	0.83	0.32 $\pm$ 0.19	0.40
IMDB	0	0.44 $\pm$ 0.06	1.00	0.67 $\pm$ 0.05	1.00	0.34 $\pm$ 0.07	1.00
	1	0.43 $\pm$ 0.06	1.00	0.68 $\pm$ 0.05	1.00	0.34 $\pm$ 0.07	0.99
ADR Tweets	0	0.47 $\pm$ 0.18	0.76	0.73 $\pm$ 0.13	0.96	0.29 $\pm$ 0.20	0.44
	1	0.49 $\pm$ 0.15	0.85	0.72 $\pm$ 0.12	0.97	0.44 $\pm$ 0.16	0.74
20News	0	0.07 $\pm$ 0.17	0.37	0.79 $\pm$ 0.07	1.00	0.06 $\pm$ 0.15	0.29
	1	0.21 $\pm$ 0.22	0.61	0.75 $\pm$ 0.08	1.00	0.20 $\pm$ 0.20	0.62
AG News	0	0.36 $\pm$ 0.13	0.82	0.78 $\pm$ 0.07	1.00	0.30 $\pm$ 0.13	0.69
	1	0.42 $\pm$ 0.13	0.90	0.76 $\pm$ 0.07	1.00	0.43 $\pm$ 0.14	0.91
Diabetes	0	0.42 $\pm$ 0.05	1.00	0.75 $\pm$ 0.02	1.00	0.41 $\pm$ 0.05	1.00
	1	0.40 $\pm$ 0.05	1.00	0.75 $\pm$ 0.02	1.00	0.45 $\pm$ 0.05	1.00
Anemia	0	0.47 $\pm$ 0.05	1.00	0.77 $\pm$ 0.02	1.00	0.46 $\pm$ 0.05	1.00
	1	0.46 $\pm$ 0.06	1.00	0.77 $\pm$ 0.03	1.00	0.47 $\pm$ 0.06	1.00
CNN	Overall	0.24 $\pm$ 0.07	0.99	0.50 $\pm$ 0.10	1.00	0.20 $\pm$ 0.07	0.98
bAbI 1	Overall	0.25 $\pm$ 0.16	0.55	0.72 $\pm$ 0.12	0.99	0.16 $\pm$ 0.14	0.28
bAbI 2	Overall	-0.02 $\pm$ 0.14	0.27	0.68 $\pm$ 0.06	1.00	-0.01 $\pm$ 0.13	0.27
bAbI 3	Overall	0.24 $\pm$ 0.11	0.87	0.61 $\pm$ 0.13	1.00	0.26 $\pm$ 0.10	0.89
SNLI	0	0.31 $\pm$ 0.23	0.36	0.59 $\pm$ 0.18	0.80	0.16 $\pm$ 0.26	0.20
	1	0.33 $\pm$ 0.21	0.38	0.58 $\pm$ 0.19	0.80	0.36 $\pm$ 0.19	0.44
	2	0.31 $\pm$ 0.21	0.36	0.57 $\pm$ 0.19	0.80	0.34 $\pm$ 0.20	0.40

The table demonstrates that:

Attention weights in BiLSTM encoders correlate weakly and inconsistently with both gradient attributions (mean $\tau_g \le 0.49$ ) and feature erasure attributions (mean $\tau_{\text{loo}} \le 0.47$ ). The fraction of statistically significant correlations ("Sig. Frac.") is low for shorter sequences.
In non-recurrent "Average" embedding encoders (where contextual token mixing is absent), attention correlates substantially more strongly with gradient measures (mean $\tau_g \approx 0.50 - 0.79$ ) and LOO measures (averaging $\sim 0.375$ higher correlation than BiLSTM).
Gradient and LOO measures correlate with each other on average $>0.20$ (for LOO) and $\sim 0.25$ (for gradients) more strongly than either correlates with BiLSTM attention weights.

Knowl 6 — Attention Weight Permutation Test for Prediction Sensitivity
algorithm
To evaluate the sensitivity of attentive models to arbitrary alterations in token weights, attention distributions are randomly shuffled across token positions while holding encoder hidden states constant.
Input: Input sequence $x$ , Query $Q$ , Encoder $\text{Enc}$ , Attention function $\phi$ , Decoder $\text{Dec}$ , Permutation iterations $P = 100$
Output: Median prediction shift $\Delta \hat{y}^{\text{med}}$
$h \leftarrow \text{Enc}(x)$
$\hat{\alpha} \leftarrow \text{softmax}(\phi(h, Q))$
$\hat{y} \leftarrow \text{Dec}(h, \hat{\alpha})$
for $p \leftarrow 1$ to $P$ do
     $\alpha^p \leftarrow \text{Permute}(\hat{\alpha})$
     $\hat{y}^p \leftarrow \text{Dec}(h, \alpha^p)$
     $\Delta \hat{y}^p \leftarrow \text{TVD}[\hat{y}^p, \hat{y}]$
end for
$\Delta \hat{y}^{\text{med}} \leftarrow \text{Median}_{p}(\Delta \hat{y}^p)$
return \Delta \hat{y}^{\text{med}}
Evaluating this procedure across benchmarks reveals that many input instances with large maximum attention values (e.g., $\max(\hat{\alpha}) > 0.5$ ) exhibit minimal median output divergence ( $\Delta \hat{y}^{\text{med}} \approx 0$ ). This demonstrates that the specific assignment of attention weights often exerts minimal impact on the final decision when contextualized representations are used.
Knowl 7 — Attentive Encoder-Decoder Neural Network Architecture
model/method
The standard attentive neural NLP architecture for sequence classification, Question Answering (QA), and Natural Language Inference (NLI) maps a sequence of one-hot token representations into a categorical output distribution via an encoder, an attention layer, and a dense classification decoder.
1. Input and Embedding: An input sequence of $T$ tokens $x \in \mathbb{R}^{T \times |V|}$ over vocabulary $V$ is mapped via embedding matrix $E$ to dense representations $x_e \in \mathbb{R}^{T \times d}$ .
2. Context Encoder: An encoder module produces hidden representations $h = \text{Enc}(x_e) \in \mathbb{R}^{T \times m}$ $h = Enc (x_{e}) \in R^{T \times m}$ . Encoders analyzed include:
  - Bidirectional LSTM (BiLSTM): Forward and backward recurrent states are concatenated.
  - CNN: Multiple 1D convolutional filter sizes (e.g., widths $[1, 3, 5, 7]$ with 64 filters each) followed by ReLU activations.
  - Average Projection: Token embeddings passed through a linear projection matrix followed by a ReLU activation (non-recurrent, unordered).
3. Attention Mechanism: Given hidden states $h$ $h$ and optional query vector $Q \in \mathbb{R}^m$ $Q \in R^{m}$ (e.g., question embedding in QA or hypothesis representation in NLI), similarity scores $\phi(h, Q)$ $ϕ (h, Q)$ are mapped to attention weights $\hat{\alpha} \in \mathbb{R}^T$ $\overset{α}{^} \in R^{T}$ :
  - Additive (Tanh) Attention: $\phi(h, Q) = v^T \tanh(W_1 h + W_2 Q)$
  - Scaled Dot-Product Attention: $\phi(h, Q) = \frac{h Q}{\sqrt{m}}$
  - Normalized Weights: $\hat{\alpha} = \text{softmax}(\phi(h, Q))$
4. Context Aggregation and Prediction: A weighted context vector $h_\alpha = \sum_{t=1}^T \hat{\alpha}_t h_t$ is decoded by a parameterized dense layer $\theta$ to produce class probabilities: $\hat{y} = \sigma(\theta \cdot h_\alpha) \in \mathbb{R}^{|\mathcal{Y}|}$ where $\sigma$ is the sigmoid function for binary tasks or softmax for multi-class classification.
Knowl 8 — Output Distance and Attention Divergence Metrics
definition
To quantify shifts in model predictions and differences between attention weight configurations, two bounded divergence metrics are employed:
1. Total Variation Distance (TVD): Measures the absolute probability shift between two output class distributions $\hat{y}_1, \hat{y}_2 \in \mathbb{R}^{|\mathcal{Y}|}$ over label set $\mathcal{Y}$ : $\text{TVD}(\hat{y}_1, \hat{y}_2) = \frac{1}{2} \sum_{i=1}^{|\mathcal{Y}|} |\hat{y}_{1i} - \hat{y}_{2i}|$ $\text{TVD}$ is bounded in $[0, 1]$ , where $0$ indicates identical predictions and $1$ represents non-overlapping categorical distributions.
2. Jensen-Shannon Divergence (JSD): Measures the symmetric divergence between two categorical attention distributions $\alpha_1, \alpha_2 \in \Delta^{T-1}$ of length $T$ : $\text{JSD}(\alpha_1, \alpha_2) = \frac{1}{2} \text{KL}\left[\alpha_1 \,\middle\|\, \frac{\alpha_1 + \alpha_2}{2}\right] + \frac{1}{2} \text{KL}\left[\alpha_2 \,\middle\|\, \frac{\alpha_1 + \alpha_2}{2}\right]$ For discrete distributions using natural logarithms, $\text{JSD}$ is bounded in $[0, \ln(2)] \approx [0, 0.693]$ , independent of sequence length $T$ .
Knowl 9 — Methodological Limitations in Evaluating Attention as Explanation
limitation
The empirical interrogation of attention mechanisms as model explanations carries several explicit methodological limitations:
1. Imperfect Ground Truth for Feature Importance: Gradient attributions and leave-one-out erasures provide well-defined, model-intrinsic semantics, but they do not constitute an absolute ground truth for human-interpretable feature importance in non-linear models.
2. Likelihood of Adversarial Configurations: Adversarially constructed attention weights are constrained only to the simplex $\Delta^{T-1}$ ; they are not constrained to be probable under the parameterized attention function $P(\alpha \mid x, Q; \theta)$ , meaning the model itself might not generate them during normal inference.
3. Multiplicity of Explanations: The existence of multiple alternative attention configurations yielding the same output could reflect true feature redundancy in the text (multiple sufficient rationales), which complicates single-attribution interpretability.
4. Model Architecture Scope: The analysis is bounded to single-layer attention over sequence representations in classification, QA, and NLI tasks with fixed label spaces, excluding multi-head co-attention networks and structured sequence-to-sequence generation tasks such as machine translation.

Coverage note — None was omitted.

References

1.David Alvarez-Melis and Tommi S Jaakkola. 2017. A causal framework for explaining the predictions of black-box sequence-to-sequence models. arXiv preprint arXiv:1707.01943.
2.Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
3.Yujia Bao, Shiyu Chang, Mo Yu, and Regina Barzilay. 2018. Deriving machine attention from human rationales. arXiv preprint arXiv:1808.09367.
4.Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326.
5.Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. 2016. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. In Advances in Neural Information Processing Systems, pages 3504–3512.
6.Shi Feng, Eric Wallace, Alvin Grissom II, Pedro Rodriguez, Mohit Iyyer, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretation difficult. In Empirical Methods in Natural Language Processing.
7.Reza Ghaeini, Xiaoli Fern, and Prasad Tadepalli. 2018. Interpreting recurrent and attention-based neural models: a case study on natural language inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4952–4957.
8.Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693–1701.
9.Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimic-iii, a freely accessible critical care database. Scientific data, 3:160035.
10.Yoon Kim, Carl Denton, Luong Hoang, and Alexander M Rush. 2017. Structured attention networks. arXiv preprint arXiv:1702.00887.
11.Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
12.Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. arXiv preprint arXiv:1606.04155.
13.Tao Lei et al. 2017. Interpretable neural models for natural language processing. Ph.D. thesis, Massachusetts Institute of Technology.
14.Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220.
15.Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490.
16.Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
17.Andre Martins and Ramon Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International Conference on Machine Learning, pages 1614–1623.
18.Azadeh Nikfarjam, Abeed Sarker, Karen OâA˘ ZConnor, Rachel Ginn, and Graciela Gonzalez. 2015. Pharmacovigilance from social media: mining adverse drug reaction mentions using sequence labeling with word embedding cluster features. Journal of the American Medical Informatics Association, 22(3):671–681.
19.Nikolaos Pappas and Andrei Popescu-Belis. 2016. Human versus machine attention in document classification: A dataset with crowdsourced annotations. In Proceedings of The Fourth International Workshop on Natural Language Processing for Social Media, pages 94–100.
20.Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. arXiv preprint arXiv:1606.01933.
21.Ben Peters, Vlad Niculae, and André FT Martins. 2018. Interpretable structure induction via sparse attention. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 365–367.
22.Andrew Slavin Ross, Michael C Hughes, and Finale Doshi-Velez. 2017. Right for the right reasons: Training differentiable models by constraining their explanations. arXiv preprint arXiv:1703.03717.
23.Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603.
24.Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642.
25.Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. arXiv preprint arXiv:1703.01365.
26.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008.
27.Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698.
28.Qizhe Xie, Xuezhe Ma, Zihang Dai, and Eduard Hovy. 2017. An interpretable knowledge transfer model for knowledge base completion. arXiv preprint arXiv:1704.05908.
29.Caiming Xiong, Victor Zhong, and Richard Socher. 2016. Dynamic coattention networks for question answering. arXiv preprint arXiv:1611.01604.
30.Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pages 2048–2057.
31.Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pages 649–657.
32.Ye Zhang, Iain Marshall, and Byron C Wallace. 2016. Rationale-augmented convolutional neural networks for text classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), volume 2016, pages 795–804.

Citation

MLA

Jain, S., and B. C. Wallace. “Attention Is Not Explanation”. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 3543–56, https://doi.org/10.18653/v1/N19-1357.

APA

Jain, S., & Wallace, B. C. (2019). Attention is not Explanation. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 3543–3556. https://doi.org/10.18653/v1/N19-1357

Chicago

Jain, S., and B. C. Wallace. 2019. “Attention Is Not Explanation”. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 3543–56. https://doi.org/10.18653/v1/N19-1357.

Harvard

Jain, S. and Wallace, B.C. (2019) “Attention is not Explanation”, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, pp. 3543–3556. Available at: https://doi.org/10.18653/v1/N19-1357.

Vancouver

1. Jain S, Wallace BC (2019) Attention is not Explanation. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, pp 3543–3556

BibTeX

@inproceedings{jain-wallace-2019-attention,
    title = "{A}ttention is not {E}xplanation",
    author = "Jain, Sarthak  and
      Wallace, Byron C.",
    editor = "Burstein, Jill  and
      Doran, Christy  and
      Solorio, Thamar",
    booktitle = "Proceedings of the 2019 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)",
    month = jun,
    year = "2019",
    address = "Minneapolis, Minnesota",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/N19-1357/",
    doi = "10.18653/v1/N19-1357",
    pages = "3543--3556"
}

Metadata:ACL Anthology

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Attention is not Explanation

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Adversarial Attention Optimization Formulation

Knowl 2 — Search Algorithm for Adversarial Attention Distributions

Knowl 3 — Empirical Invariance of Predictions Under Adversarial Attention Configurations

Knowl 4 — Disconnected-Graph Gradient and Leave-One-Out Feature Importance Estimation

Knowl 5 — Correlation of Attention Weights with Gradient and Leave-One-Out Importance Measures

Knowl 6 — Attention Weight Permutation Test for Prediction Sensitivity

Knowl 7 — Attentive Encoder-Decoder Neural Network Architecture

Knowl 8 — Output Distance and Attention Divergence Metrics

Knowl 9 — Methodological Limitations in Evaluating Attention as Explanation

References

Citation

BibTeX

Source Code

Access the Paper