Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

bias injection

Bias injection is a machine learning process that introduces specific structural assumptions, domain priors, or learning constraints into a model to guide its pattern recognition and improve generalization. Models with minimal built-in assumptions often require immense amounts of training data to discover fundamental patterns independently, making bias injection useful for constraining the hypothesis space and enabling more effective learning from limited datasets. This technique is typically achieved through specialized architectural components, tailored loss functions, or knowledge distillation frameworks where a recipient model acquires desirable structural preferences, such as spatial locality or relational awareness, from specialized teacher models or structured priors.

1 item

Co-advise: Cross Inductive Bias Distillation

Co-advise: Cross Inductive Bias Distillation

Sucheng Ren, Zhengqi Gao, Tianyu Hua, Zihui Xue, Yonglong Tian, Shengfeng He, Hang Zhao

OrganizationsMassachusetts Institute of TechnologyShanghai Qi Zhi InstituteSouth China University of TechnologyTsinghua UniversityUniversity of Texas at Austin

Why you should read this

Demonstrates that distilling knowledge from multiple lightweight teachers with complementary inductive biases, such as convolution and involution, boosts vision transformer performance beyond traditional heavy-teacher distillation while reducing computational costs.

The inductive bias of vision transformers is more relaxed that cannot work well with insufficient data. Knowledge distillation is thus introduced to assist the training of transformers. Unlike previous works, where merely heavy convolution-based teachers are provided, in this paper, we delve into the influence of models inductive biases in knowledge distillation (e.g., convolution and involution). Our key observation is that the teacher accuracy is not the dominant reason for the student accuracy, but the teacher inductive bias is more important. We demonstrate that lightweight teachers with different architectural inductive biases can be used to co-advise the student transformer with outstanding performances. The rationale behind is that models designed with different inductive biases tend to focus on diverse patterns, and teachers with different inductive biases attain various knowledge despite being trained on the same dataset. The diverse knowledge provides a more precise and comprehensive description of the data and compounds and boosts the performance of the student during distillation. Furthermore, we propose a token inductive bias alignment to align the inductive bias of the token with its target teacher model. With only lightweight teachers provided and using this cross inductive bias distillation method, our vision transformers (termed as CiT) outperform all previous vision transformers (ViT) of the same architecture on ImageNet. Moreover, our small size model CiT-SAK further achieves 82.7% Top-1 accuracy on ImageNet without modifying the attention module of the ViT. Code is available at https://github.com/OliverRensu/co-advise.

Added

2026-09-26