Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

cross inductive bias distillation

Cross inductive bias distillation is a knowledge distillation technique in machine learning where a student model is trained using guidance from teacher models that feature different architectural inductive biases. Because neural network architectures with distinct structural priors, such as convolutions or involutions, attend to diverse patterns and extract complementary representations from identical data, transferring knowledge across these different biases exposes the student to a broader range of feature perspectives. This framework is primarily applied to enhance models with more relaxed inductive biases, such as vision transformers, by aligning and transferring heterogeneous structural knowledge from specialized teachers to improve student model accuracy, data efficiency, and generalization.

1 item

Co-advise: Cross Inductive Bias Distillation

Co-advise: Cross Inductive Bias Distillation

Sucheng Ren, Zhengqi Gao, Tianyu Hua, Zihui Xue, Yonglong Tian, Shengfeng He, Hang Zhao

OrganizationsMassachusetts Institute of TechnologyShanghai Qi Zhi InstituteSouth China University of TechnologyTsinghua UniversityUniversity of Texas at Austin

Why you should read this

Demonstrates that distilling knowledge from multiple lightweight teachers with complementary inductive biases, such as convolution and involution, boosts vision transformer performance beyond traditional heavy-teacher distillation while reducing computational costs.

The inductive bias of vision transformers is more relaxed that cannot work well with insufficient data. Knowledge distillation is thus introduced to assist the training of transformers. Unlike previous works, where merely heavy convolution-based teachers are provided, in this paper, we delve into the influence of models inductive biases in knowledge distillation (e.g., convolution and involution). Our key observation is that the teacher accuracy is not the dominant reason for the student accuracy, but the teacher inductive bias is more important. We demonstrate that lightweight teachers with different architectural inductive biases can be used to co-advise the student transformer with outstanding performances. The rationale behind is that models designed with different inductive biases tend to focus on diverse patterns, and teachers with different inductive biases attain various knowledge despite being trained on the same dataset. The diverse knowledge provides a more precise and comprehensive description of the data and compounds and boosts the performance of the student during distillation. Furthermore, we propose a token inductive bias alignment to align the inductive bias of the token with its target teacher model. With only lightweight teachers provided and using this cross inductive bias distillation method, our vision transformers (termed as CiT) outperform all previous vision transformers (ViT) of the same architecture on ImageNet. Moreover, our small size model CiT-SAK further achieves 82.7% Top-1 accuracy on ImageNet without modifying the attention module of the ViT. Code is available at https://github.com/OliverRensu/co-advise.

Added

2026-09-26