Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

training loss

Training loss is a metric in machine learning that measures the magnitude of error between a model predictions and the actual target values on the dataset used during the learning process. Calculated through a mathematical loss function, such as cross-entropy or mean squared error, it quantifies how accurately the model currently fits the training data. Optimization algorithms use the gradients of this loss to iteratively adjust the internal parameters of the model, aiming to minimize the error over successive training steps. While a declining training loss indicates that the model is successfully learning patterns from the training inputs, tracking it in relation to validation performance is necessary to ensure the model generalizes effectively rather than overfitting to the training data.

1 item

Understanding Emergent Abilities of Language Models from the Loss Perspective

Understanding Emergent Abilities of Language Models from the Loss Perspective

Zhengxiao Du, Aohan Zeng, Yuxiao Dong, Jie Tang

OrganizationsTsinghua UniversityZhipu AI

Why you should read this

Demonstrates that pre-training loss reliably predicts downstream task performance across different model and data sizes, revealing that emergent abilities consistently appear only after loss drops below sharp, metric-independent thresholds.

Recent studies have put into question the belief that emergent abilities [58] in language models are exclusive to large models. This skepticism arises from two observations: 1) smaller models can also exhibit high performance on emergent abilities and 2) there is doubt on the discontinuous metrics used to measure these abilities. In this paper, we propose to study emergent abilities in the lens of pre-training loss, instead of model size or training compute. We demonstrate that the Transformer models with the same pre-training loss, but different model and data sizes, generate the same performance on various downstream tasks, with a fixed data corpus, tokenization, and model architecture. We also discover that a model exhibits emergent abilities on certain tasks—regardless of the continuity of metrics—when its pre-training loss falls below a specific threshold. Before reaching this threshold, its performance remains at the level of random guessing. This inspires us to redefine emergent abilities as those that manifest in models with lower pre-training losses, highlighting that these abilities cannot be predicted by merely extrapolating the performance trends of models with higher pre-training losses.

Added

2026-09-26