Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

inductive bias distillation

Inductive bias distillation is a machine learning method in knowledge distillation where a student model is trained to acquire the structural assumptions, or inductive biases, inherent in one or more teacher architectures. Unlike conventional distillation that primarily focuses on transferring raw predictive accuracy from larger networks, this approach deliberately transfers architectural priors, such as spatial locality or translation equivariance, from models embedded with strong domain constraints to more flexible, low-bias models like vision transformers. By absorbing the distinct structural patterns captured by these specialized teachers, the student network compensates for its own lack of built-in architectural constraints, resulting in improved data efficiency, feature representation, and generalization without requiring massive training datasets.

1 item

Co-advise: Cross Inductive Bias Distillation

Co-advise: Cross Inductive Bias Distillation

Sucheng Ren, Zhengqi Gao, Tianyu Hua, Zihui Xue, Yonglong Tian, Shengfeng He, Hang Zhao

OrganizationsMassachusetts Institute of TechnologyShanghai Qi Zhi InstituteSouth China University of TechnologyTsinghua UniversityUniversity of Texas at Austin

Why you should read this

Demonstrates that distilling knowledge from multiple lightweight teachers with complementary inductive biases, such as convolution and involution, boosts vision transformer performance beyond traditional heavy-teacher distillation while reducing computational costs.

The inductive bias of vision transformers is more relaxed that cannot work well with insufficient data. Knowledge distillation is thus introduced to assist the training of transformers. Unlike previous works, where merely heavy convolution-based teachers are provided, in this paper, we delve into the influence of models inductive biases in knowledge distillation (e.g., convolution and involution). Our key observation is that the teacher accuracy is not the dominant reason for the student accuracy, but the teacher inductive bias is more important. We demonstrate that lightweight teachers with different architectural inductive biases can be used to co-advise the student transformer with outstanding performances. The rationale behind is that models designed with different inductive biases tend to focus on diverse patterns, and teachers with different inductive biases attain various knowledge despite being trained on the same dataset. The diverse knowledge provides a more precise and comprehensive description of the data and compounds and boosts the performance of the student during distillation. Furthermore, we propose a token inductive bias alignment to align the inductive bias of the token with its target teacher model. With only lightweight teachers provided and using this cross inductive bias distillation method, our vision transformers (termed as CiT) outperform all previous vision transformers (ViT) of the same architecture on ImageNet. Moreover, our small size model CiT-SAK further achieves 82.7% Top-1 accuracy on ImageNet without modifying the attention module of the ViT. Code is available at https://github.com/OliverRensu/co-advise.

Added

2026-09-26