Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

Transformer

A Transformer is a deep learning model architecture designed to process sequential data, such as text, by relying on a self-attention mechanism to weigh the relationships between all elements in a sequence simultaneously. Unlike earlier recurrent architectures that processed inputs step by step, the Transformer analyzes entire sequences in parallel, significantly improving computational efficiency and scalability during training. The core architecture consists of stacked layers featuring multi-head self-attention, position-wise feed-forward networks, residual connections, and layer normalization, which together enable the model to capture complex contextual dependencies across long distances. While initially developed for natural language processing tasks such as machine translation, the Transformer has become the foundational framework for modern large language models, computer vision systems, and multimodal artificial intelligence.

1 item

Measuring the Mixing of Contextual Information in the Transformer

Measuring the Mixing of Contextual Information in the Transformer

Javier Ferrando, Gerard I. Gállego, Marta R. Costa-jussà

OrganizationsUniversitat Politècnica de Catalunya

Why you should read this

Proposes ALTI, an interpretability method that tracks information flow across full Transformer attention blocks to generate input attributions that surpass gradient-based techniques in faithfulness and reliability.

The Transformer architecture aggregates input information through the self-attention mechanism, but there is no clear understanding of how this information is mixed across the entire model. Additionally, recent works have demonstrated that attention weights alone are not enough to describe the flow of information. In this paper, we consider the whole attention block –multi-head attention, residual connection, and layer normalization– and define a metric to measure token-to-token interactions within each layer. Then, we aggregate layer-wise interpretations to provide input attribution scores for model predictions. Experimentally, we show that our method, ALTI (Aggregation of Layer-wise Token-to-token Interactions), provides more faithful explanations and increased robustness than gradient-based methods.

Added

2026-09-26