Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

mask search

Mask search is an optimization process in neural network pruning used to identify an optimal selection mask that determines which network components, such as individual weights, channels, or attention heads, should be retained or removed. Rather than discarding parameters arbitrarily, a mask search evaluates the importance or sensitivity of different model components based on statistical, gradient, or informational metrics to find a sparse subnetwork configuration. This search operates under specified computational or memory budgets, enabling deep learning models to reduce latency and resource overhead while preserving baseline accuracy and task performance.

1 item

A Fast Post-Training Pruning Framework for Transformers

A Fast Post-Training Pruning Framework for Transformers

Woosuk Kwon, Sehoon Kim, Michael W. Mahoney, Joseph Hassoun, Kurt Keutzer, Amir Gholami

OrganizationsInternational Computer Science InstituteLawrence Berkeley National LaboratorySamsung Semiconductor, Inc.University of California Berkeley

Why you should read this

Proposes a retraining-free structured pruning framework for Transformers that achieves up to a 2.0x reduction in FLOPs with under 1% accuracy loss in less than three minutes on a single GPU.

Pruning is an effective way to reduce the huge inference cost of Transformer models. However, prior work on pruning Transformers requires retraining the models. This can add high training cost and high complexity to model deployment, making it difficult to use in many practical situations. To address this, we propose a fast post-training pruning framework for Transformers that does not require any retraining. Given a resource constraint and a sample dataset, our framework automatically prunes the Transformer model using structured sparsity methods. To retain high accuracy without retraining, we introduce three novel techniques: (i) a lightweight mask search algorithm that finds which heads and filters to prune based on the Fisher information; (ii) mask rearrangement that complements the search algorithm; and (iii) mask tuning that reconstructs the output activations for each layer. We apply our method to BERT_BASE and DistilBERT, and we evaluate its effectiveness on GLUE and SQuAD benchmarks. Our framework achieves up to 2.0× reduction in FLOPs and 1.56× speedup in inference latency, while maintaining < 1% loss in accuracy. Importantly, our framework prunes Transformers in less than 3 minutes on a single GPU, which is over two orders of magnitude faster than existing pruning approaches that retrain the models.1

Added

2026-09-26