Bridging the Gap Between Anchor-Based and Anchor-Free Detection via Adaptive Training Sample Selection
Shifeng ZhangCheng ChiYongqiang YaoZhen LeiStan Z. Li
Reveals that positive and negative sample selection is the essential difference between anchor-based and anchor-free object detectors, introducing an Adaptive Training Sample Selection (ATSS) strategy that unifies both paradigms and boosts detection accuracy without added overhead.
Modern computer vision systems rely heavily on object detection models to identify and locate items in images for applications like visual recognition and surveillance. For years, the field was divided into anchor-based systems, which predict object boundaries starting from preset reference boxes, and anchor-free systems, which detect objects starting from key points. While anchor-free systems recently demonstrated superior performance, the technical reasons driving this performance gap remained misunderstood.
The article set out to determine the fundamental root cause of the performance disparity between anchor-based and anchor-free detectors and to develop an automated method to bridge that gap.
To conduct a fair evaluation, the authors aligned implementation details—such as normalization and loss functions—between representative anchor-based (RetinaNet) and anchor-free (FCOS) frameworks using the standard MS COCO dataset of 80 object classes. By isolating and testing the remaining differences, the researchers investigated whether the regression starting point (box versus point) or the sample selection strategy (how training data is classified as positive or negative) drove model accuracy.
The findings reveal that the starting status—whether predicting from an anchor point or an anchor box—has virtually no effect on final accuracy (yielding identical 37.8% Average Precision scores under matched conditions). Instead, the essential driver of performance is how models define positive (foreground) and negative (background) training samples. In response, the authors introduced Adaptive Training Sample Selection (ATSS), an automated mechanism that calculates dynamic thresholds based on the mean and standard deviation of candidate sample overlap. When applied to standard detectors, ATSS improved RetinaNet by 2.3% and FCOS by 1.4% without adding computational overhead. Furthermore, experiments demonstrated that placing multiple preset anchor boxes per location becomes entirely unnecessary once sample selection is optimized. Combined with advanced backbones and testing strategies, ATSS achieved a state-of-the-art 50.7% Average Precision.
These results show that system designers do not need to choose between anchor-based and anchor-free designs based on perceived accuracy differences, nor do they need to waste computational resources managing complex, dense anchor grids. By eliminating sensitive manual tuning parameters (such as rigid overlap thresholds), ATSS simplifies detector design, lowers engineering maintenance, and boosts overall model accuracy at zero extra inference cost.
Development teams building or deploying vision systems should adopt adaptive sample selection strategies and streamline architectures to use single presets per location rather than dense multi-anchor grids. Organizations should also consider incorporating ATSS into existing computer vision pipelines to capture immediate performance gains. Future work should further explore the underlying role and utility of multiple anchors under adaptive frameworks.
Confidence in these findings is high given the strict experimental controls and validated benchmarks across multiple model architectures on the MS COCO dataset. However, stakeholders should note that the evaluations primarily focus on standard one-stage and center-based detection frameworks, meaning custom or atypical architectures should be independently piloted before broad implementation.
- Paper: Focal Loss for Dense Object Detection, Tsung-Yi Lin et al. (2017). Reading Focal Loss for Dense Object Detection first is essential because the source paper builds directly upon RetinaNet's backbone and focal loss formulation to investigate training sample selection.
- Paper: FCOS: Fully Convolutional One-Stage Object Detection, Zhi Tian et al. (2019). FCOS provides the foundational anchor-free detection framework whose positive and negative sample definitions are directly analyzed and compared against anchor-based methods in the source paper.
- Paper: YOLOX: Exceeding YOLO Series in 2021, Zheng Ge et al. (2021). YOLOX directly extends the adaptive sample selection principles of ATSS by incorporating them into a high-performance modern real-time detector.
- Paper: Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection, Shilong Liu et al. (2023). Grounding DINO builds upon advanced detection architectures to extend closed-set detectors into open-vocabulary settings, making the source paper's training sample strategies vital preparatory reading.
