Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentation

Zhaohui ZhengPing WangDongwei RenWei LiuRongguang YeQinghua HuWangmeng Zuo

article2020IEEE Transactions on Cybernetics1,415 citationsESI Highly Cited Paper

Proposes Complete-IoU loss and Cluster-NMS to integrate key geometric factors into bounding box regression and suppression, boosting object detection and instance segmentation accuracy across popular architectures without sacrificing real-time inference speed.

Listen

Modern computer vision systems rely heavily on object detection and instance segmentation for applications ranging from autonomous navigation to video surveillance. These systems locate targets using predicted bounding boxes, but traditional methods struggle in two key stages: model training often uses loss functions that fail to distinguish complex spatial alignments or converge too slowly, and model inference relies on post-processing techniques, known as non-maximum suppression, that create computational bottlenecks or mistakenly remove overlapping objects.

The main objective of the article is to demonstrate that incorporating three fundamental geometric factors—overlap area, normalized central point distance, and aspect ratio—into both model training and post-processing significantly improves accuracy and recall without sacrificing real-time inference speed.

To achieve this, the authors developed Complete Intersection over Union (CIoU) loss for model training and Cluster-NMS for fast inference. The approach was evaluated through extensive numerical simulations covering over 1.7 million spatial configurations, followed by empirical validation across benchmark datasets (MS COCO and PASCAL VOC) using widely adopted deep learning architectures, including YOLACT, BlendMask-RT, YOLO v3, SSD, and Faster R-CNN.

The investigation produced several key findings. First, CIoU loss significantly speeds up model convergence and achieves superior accuracy compared to standard norm-based and prior overlap-based losses, yielding gains such as a 5.67% increase in average precision on YOLO v3 over baseline methods. Second, Cluster-NMS operates via implicit box clustering executed entirely on graphics processing units (GPUs), matching the exact suppression output of original sequential methods while requiring fewer iterations. Third, combining Cluster-NMS with geometric enhancements (such as score penalties and coordinate weighting) produced substantial accuracy improvements—for example, boosting YOLACT's detection average precision by +1.7 and average recall by +6.2 on the MS COCO benchmark while sustaining a real-time speed of 27.1 frames per second. Finally, the analysis showed that Cluster-NMS variants deliver weighted-combination accuracy at more than six times the computational speed of previous weighted methods.

These findings imply that vision pipelines can achieve notable accuracy and safety improvements in crowded or occluded environments without requiring costly hardware upgrades or slower processing times. Because CIoU loss and Cluster-NMS act as plug-and-play components, organizations can upgrade existing vision systems with minimal engineering friction.

Engineering teams should deploy CIoU loss during model training and adopt Cluster-NMS variants during deployment to maximize both precision and recall. Depending on specific operational needs, practitioners can tune the trade-off parameter to prioritize precision or recall, using score-penalty variants for dense segmentation tasks and weighted-coordinate variants for rapid object detection.

The primary limitation identified is that incorporating aspect ratio can slightly reduce performance on small or medium objects, where center-point distance dominates spatial alignment. Future research should refine aspect ratio penalties for smaller targets and explore further low-level CUDA optimizations for Cluster-NMS implementations.

Harbin Institute of Technology Tianjin University clustering algorithms convolutional neural networks deep learning instance segmentation object detection geometric factors

arXiv: 2005.03572 Zzh-tju/CIoU

Paper: Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression, Zhaohui Zheng et al. (2019). Introduces the Distance-IoU (DIoU) and Complete-IoU (CIoU) formulations for bounding box regression, establishing the fundamental geometric penalty terms directly extended and analyzed in this work.
Paper: Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression, Hamid Rezatofighi et al. (2019). Derives Generalized IoU (GIoU) loss to handle non-overlapping bounding boxes, laying the essential groundwork for geometric IoU-based loss functions.
Paper: UnitBox: An Advanced Object Detection Network, Jiahui Yu et al. (2016). Proposes the foundational IoU loss for bounding box regression, motivating direct overlap optimization over traditional coordinate-wise Ln-norm losses.
Paper: YOLACT: Real-Time Instance Segmentation, Daniel Bolya et al. (2019). Presents real-time instance segmentation with Fast NMS, serving as a primary target architecture and post-processing benchmark improved by Cluster-NMS and CIoU.
Paper: Soft-NMS — Improving Object Detection with One Line of Code, Navaneeth Bodla et al. (2017). Addresses the limitations of standard greedy NMS in dense scenes via continuous score decay, motivating the design of faster and more geometric post-processing suppression algorithms.
Paper: Mask R-CNN, Kaiming He et al. (2017). Establishes the standard baseline framework for simultaneous object detection and instance segmentation evaluated throughout the paper.
Paper: YOLOv3: An Incremental Improvement, Joseph Redmon et al. (2018). Provides a prominent one-stage real-time object detection architecture used as an experimental benchmark to demonstrate the efficacy of CIoU and Cluster-NMS.
Paper: Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, Shaoqing Ren et al. (2015). Introduces the canonical two-stage anchor-based object detection pipeline and standard bounding-box regression formulation.

Paper: Focal and Efficient IOU Loss for Accurate Bounding Box Regression, Yi-Fan Zhang et al. (2021). Builds directly on CIoU by explicitly decoupling aspect ratio into width and height error terms (EIoU) and adding focal sample reweighting to resolve gradient conflict issues.
Paper: YOLOv4: Optimal Speed and Accuracy of Object Detection, Alexey Bochkovskiy et al. (2020). Adopts CIoU loss as a core 'bag-of-freebies' component to significantly boost training convergence and localization accuracy in real-time detectors.
Paper: Scaled-YOLOv4: Scaling Cross Stage Partial Network, Chien-Yao Wang et al. (2020). Scales the YOLOv4 architecture across multiple hardware platforms while leveraging CIoU loss and enhanced NMS techniques for improved real-time performance.
Paper: Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection, Xiang Li et al. (2020). Extends geometric bounding box regression by formulating box offsets as continuous probability distributions to capture boundary ambiguity.
Paper: TOOD: Task-aligned One-stage Object Detection, Chengjian Feng et al. (2021). Aligns classification and IoU-based localization objectives during training and inference to reduce the post-processing suppression errors analyzed in Cluster-NMS.
Paper: YOLOv6: A Single-Stage Object Detection Framework for Industrial Applications, Chuyi Li et al. (2022). Incorporates modern geometric regression losses and post-processing strategies into a production-focused real-time detection framework.
Paper: YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors, Chien-Yao Wang et al. (2023). Integrates trainable optimization techniques and advanced regression objectives to advance real-time object detection accuracy.
Paper: A Comprehensive Review of YOLO Architectures in Computer Vision: From YOLOv1 to YOLOv8 and YOLO-NAS, Juan R. Terven et al. (2023). Provides a comprehensive historical overview tracking how geometric IoU losses and NMS refinements became standardized across the YOLO family.

Cover for Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentation

Abstract

Deep learning-based object detection and instance segmentation have achieved unprecedented progress. In this paper, we propose Complete-IoU (CIoU) loss and Cluster-NMS for enhancing geometric factors in both bounding box regression and Non-Maximum Suppression (NMS), leading to notable gains of average precision (AP) and average recall (AR), without the sacrifice of inference efficiency. In particular, we consider three geometric factors, i.e., overlap area, normalized central point distance and aspect ratio, which are crucial for measuring bounding box regression in object detection and instance segmentation. The three geometric factors are then incorporated into CIoU loss for better distinguishing difficult regression cases. The training of deep models using CIoU loss results in consistent AP and AR improvements in comparison to widely adopted $\ell_n$ -norm loss and IoU-based loss. Furthermore, we propose Cluster-NMS, where NMS during inference is done by implicitly clustering detected boxes and usually requires less iterations. Cluster-NMS is very efficient due to its pure GPU implementation, and geometric factors can be incorporated to improve both AP and AR. In the experiments, CIoU loss and Cluster-NMS have been applied to state-of-the-art instance segmentation (e.g., YOLACT and BlendMask-RT), and object detection (e.g., YOLO v3, SSD and Faster R-CNN) models. Taking YOLACT on MS COCO as an example, our method achieves performance gains as +1.7 AP and +6.2 AR $_{100}$ for object detection, and +0.9 AP and +3.5 AR $_{100}$ for instance segmentation, with 27.1 FPS on one NVIDIA GTX 1080Ti GPU. All the source code and trained models are available at this https URL

Topics

clustering algorithms convolutional neural networks deep learning gpu implementation inference source codes ssd network management systems object segmentation probabilistic inference

Keywords

instance segmentation object detection geometric factors box regression YOLACT model localization confidence instance mask pedestrian detection scene text detection

I Introduction
II Related Work
II-A Object Detection and Instance Segmentation
II-B Loss Function for Bounding Box Regression
II-C Non-Maximum Suppression
III Complete-IoU Loss
III-A Analysis to IoU and GIoU Losses
III-A1 Simulation Experiment
III-A2 Limitations of IoU and GIoU Losses
III-B CIoU Loss
IV Cluster-NMS
IV-A Cluster-NMS
IV-B Incorporating Geometric Factors into Cluster-NMS
IV-B1 Score Penalty Mechanism into Cluster-NMS
IV-B2 Normalized Central Point Distance into Cluster-NMS
IV-B3 Weighted Coordinates into Cluster-NMS
IV-C Theoretical Analysis
V Experimental Results
V-A Instance Segmentation
V-A1 YOLACT
V-A2 BlendMask-RT
V-B Object Detection
V-B1 YOLO v3 [9]
V-B2 SSD [10]
V-B3 Faster R-CNN [15]
V-C Discussion
VI Conclusion
References

Knowls

Knowl 1 — Complete-IoU (CIoU) Loss Formulation
equation
The Complete-IoU (CIoU) loss incorporates three geometric factors—overlap area, normalized central point distance, and aspect ratio consistency—into bounding box regression. For a predicted bounding box $B = [x, y, w, h]^T$ and a target ground-truth bounding box $B^{gt} = [x^{gt}, y^{gt}, w^{gt}, h^{gt}]^T$ , the loss function is defined as:

$\mathcal{L}_{CIoU} = 1 - \text{IoU}(B, B^{gt}) + \frac{\rho^2(p, p^{gt})}{c^2} + \alpha V$

where:
- $\text{IoU}(B, B^{gt}) = \frac{|B \cap B^{gt}|}{|B \cup B^{gt}|}$ is the Intersection over Union.
- $p = [x, y]^T$ and $p^{gt} = [x^{gt}, y^{gt}]^T$ denote the central points of boxes $B$ and $B^{gt}$ , respectively.
- $\rho(p, p^{gt})$ is the Euclidean distance $\|p - p^{gt}\|_2$ .
- $c$ is the diagonal length of the smallest enclosing bounding box $C$ covering both $B$ and $B^{gt}$ .
- $V$ measures the consistency of the aspect ratios between the two bounding boxes:
$V = \frac{4}{\pi^2} \left( \arctan\frac{w^{gt}}{h^{gt}} - \arctan\frac{w}{h} \right)^2$
- $\alpha$ is a trade-off parameter controlling the penalty for aspect ratio discrepancy:
$\alpha = \begin{cases} 0, & \text{if } \text{IoU}(B, B^{gt}) < 0.5 \\ \frac{V}{(1 - \text{IoU}(B, B^{gt})) + V}, & \text{if } \text{IoU}(B, B^{gt}) \ge 0.5 \end{cases}$

When $\text{IoU}(B, B^{gt}) < 0.5$ , $\alpha = 0$ , causing $\mathcal{L}_{CIoU}$ to degrade to the Distance-IoU (DIoU) loss. All three components— $(1 - \text{IoU})$ , $\frac{\rho^2(p, p^{gt})}{c^2}$ , and $V$ —are scale-invariant and normalized to $[0, 1]$ .
Knowl 2 — Cluster-NMS Algorithm
algorithm
Cluster-NMS performs Non-Maximum Suppression by iteratively computing parallel matrix multiplications on implicitly grouped clusters of candidate bounding boxes on GPU hardware.
Input: Detected bounding boxes $B = [B_1, B_2, \dots, B_N]^T$ sorted in descending order of classification scores $s_1 \ge s_2 \ge \dots \ge s_N$ , and IoU threshold $\varepsilon \in (0, 1)$ .
Output: Binary vector $b = [b_1, b_2, \dots, b_N] \in \{0, 1\}^N$ , where $b_i = 1$ denotes reservation and $b_i = 0$ denotes suppression.
Initialize maximum iterations $T = N$ , iteration counter $t = 1$ , and initial reservation vector $b^0 = [1, 1, \dots, 1] \in \{1\}^N$
Compute pairwise IoU matrix $X \in \mathbb{R}^{N \times N}$ with entries $X_{ij} = \text{IoU}(B_i, B_j)$
$X \leftarrow \text{triu}(X)$ with $X_{ii} = 0$ for all $i \in \{1, \dots, N\}$ (strictly upper triangular)
while $t \le T$ do
     $A^t \leftarrow \text{diag}(b^{t-1})$
     $C^t \leftarrow A^t \times X$
    for each column $j \in \{1, \dots, N\}$ do
         $g_j \leftarrow \max_{i} C^t_{ij}$
        if $g_j < \varepsilon$ then
             $b^t_j \leftarrow 1$
        else
             $b^t_j \leftarrow 0$
        end if
    end for
    if $b^t == b^{t-1}$ then
        break
    end if
     $t \leftarrow t + 1$
end while
return $b^t$
The algorithm computes an upper-triangular IoU matrix $X$ ensuring that a box $B_j$ is only suppressed by preceding higher-scoring boxes $B_i$ ( $i < j$ ). At iteration $t$ , multiplying by $A^t = \text{diag}(b^{t-1})$ zeros out rows of previously suppressed boxes so that they cannot suppress subsequent lower-scoring boxes.
Knowl 3 — Equivalence and Convergence Bound of Cluster-NMS to Sequential NMS
theoretical result
Let $B = [B_1, \dots, B_N]^T$ be a set of $N$ candidate bounding boxes sorted in non-ascending order of classification scores $s_1 \ge s_2 \ge \dots \ge s_N$ , and let $\varepsilon$ be the IoU suppression threshold.
1. Exact Equivalence: For $T = N$ iterations, the binary suppression vector $b^T$ obtained by Cluster-NMS is identical to the final output of standard sequential Greedy NMS.
2. Cluster Size Convergence Bound: Candidate bounding boxes can be partitioned into disjoint connected clusters of overlapping boxes, where a box $B_j$ belongs to a cluster $B^*$ if and only if there exists an overlapping box $B_i \in B^*$ with $\text{IoU}(B_i, B_j) \ge \varepsilon$ while having no overlap $\ge \varepsilon$ with any box outside $B^*$ . If $M$ is the number of boxes in the largest cluster ( $M \le N$ ), Cluster-NMS converges to the exact sequential NMS result in at most $M$ iterations ( $t^* \le M$ ). Because disconnected clusters are processed entirely in parallel without inter-cluster interference, Cluster-NMS terminates early once $b^t = b^{t-1}$ .
Knowl 4 — Geometric Extensions of Cluster-NMS
model/method
Cluster-NMS accommodates score penalties, normalized central point distances, and coordinate weighting mechanisms within its GPU-parallel matrix framework:
1. Score Penalty Mechanism ( $\text{Cluster-NMS}_S$ ): Instead of hard thresholding, classification score $s_j$ is penalized using a Gaussian decay over unsuppressed overlapping predecessor boxes:
$s_j = s_j \prod_{i=1}^{N} \exp\left( -\frac{(A \times X)_{ij}^2}{\sigma} \right)$

where $\sigma = 0.2$ and $A = \text{diag}(b)$ .
1. Distance-Penalized Score Mechanism ( $\text{Cluster-NMS}_{S+D}$ ): Normalized central point distance $D_{ij} = \frac{\rho^2(p_i, p_j)}{c_{ij}^2}$ is added to the penalty term to prevent over-suppression in occlusion scenarios:
$s_j = s_j \prod_{i=1}^{N} \min\left\{ \exp\left( -\frac{(A \times X)_{ij}^2}{\sigma} \right) + D_{ij}^\beta, \, 1 \right\}$

where $\beta = 0.6$ balances precision and recall, $\rho(p_i, p_j)$ is the Euclidean distance between box centers, and $c_{ij}$ is the diagonal length of the smallest enclosing box covering $B_i$ and $B_j$ .
1. Weighted Coordinates ( $\text{Cluster-NMS}_W$ and $\text{Cluster-NMS}_{W+D}$ ): Bounding box coordinates are replaced by a weighted average of candidate coordinates:
$B = \frac{C' \times B}{\text{Repmat}_4\left(\sum_{i} C'(i, :)\right)}$

where $C'_{ij} = s_i \cdot C_{ij}$ . For $\text{Cluster-NMS}_W$ , $C$ is constructed from the standard IoU matrix $X$ . For $\text{Cluster-NMS}_{W+D}$ , $X_{ij}$ incorporates normalized distance: $X_{ij} = \text{IoU}(B_i, B_j) - D_{ij}$ .
Knowl 5 — Simulation Experiment Protocol for Evaluating Bounding Box Regression Losses
experimental setup
A simulation protocol evaluates bounding box regression losses across 1,715,000 spatial configurations under controlled synthetic conditions:
- Target Boxes: 7 unit-area boxes (area = 1.0) centered at $(10, 10)$ with 7 aspect ratios ( $w:h \in \{1:4, 1:3, 1:2, 1:1, 2:1, 3:1, 4:1\}$ ).
- Anchor Locations: 5,000 anchor center points uniformly distributed in a disk of radius 3 centered at $(10, 10)$ .
- Anchor Scales and Aspect Ratios: At each anchor point, 7 scale areas $\{0.5, 0.67, 0.75, 1.0, 1.33, 1.5, 2.0\}$ and 7 aspect ratios $\{1:4, 1:3, 1:2, 1:1, 2:1, 3:1, 4:1\}$ are tested, yielding $5,000 \times 7 \times 7 = 245,000$ anchor boxes per target box ( $1,715,000$ total combinations).
- Optimization: For each regression case, the predicted box is updated over $T = 200$ iterations via gradient descent:
$B^t = B^{t-1} + \eta (2 - \text{IoU}^{t-1}) \nabla_{B^{t-1}} \mathcal{L}(B^{t-1}, B^{gt})$

with learning rate schedule $\eta = 0.1$ for $t \le 0.8T$ , $\eta = 0.01$ for $0.8T < t \le 0.9T$ , and $\eta = 0.001$ for $t > 0.9T$ .
- Evaluation: The cumulative $\ell_1$ -norm distance error between predictions and target boxes is tracked over time.

Knowl 6 — Instance Segmentation Performance of CIoU and Cluster-NMS on YOLACT

data/table

Instance segmentation evaluation of YOLACT-550 (ResNet-101-FPN) trained with $\text{Smooth-}\ell_1$ loss versus Complete-IoU (CIoU) loss, evaluated across multiple NMS post-processing strategies on the MS COCO 2017 validation set (batch size 4 per GPU on an NVIDIA GTX 1080Ti GPU):

Loss	NMS Strategy	FPS	Time (ms)	AP	$\text{AP}_{50}$	$\text{AP}_{75}$	$\text{AP}_S$	$\text{AP}_M$	$\text{AP}_L$	$\text{AR}_{100}$	$\text{AR}_L$
$\text{Smooth-}\ell_1$	Fast NMS	30.6	32.7	29.1	47.4	30.5	9.4	32.0	48.5	40.3	59.8
$\mathcal{L}_{CIoU}$	Fast NMS	30.6	32.7	29.6	48.1	30.9	9.4	32.0	49.7	40.3	60.8
$\mathcal{L}_{CIoU}$	Original NMS	11.5	86.6	29.7	48.3	31.0	9.4	32.2	49.8	41.7	62.8
$\mathcal{L}_{CIoU}$	Cluster-NMS	28.8	34.7	29.7	48.3	31.0	9.4	32.2	49.7	41.7	62.8
$\mathcal{L}_{CIoU}$	$\text{Cluster-NMS}_S$	28.6	35.0	30.3	49.1	31.7	9.7	33.0	50.8	43.6	65.9
$\mathcal{L}_{CIoU}$	$\text{Cluster-NMS}_{S+D}$	27.1	36.9	30.2	48.9	31.7	9.6	32.8	50.7	43.8	66.4

The results show that training with $\mathcal{L}_{CIoU}$ improves mask AP from 29.1 to 29.6 over $\text{Smooth-}\ell_1$ under Fast NMS. Cluster-NMS matches the exact precision and recall of Original NMS (29.7 AP, 41.7 $\text{AR}_{100}$ ) while operating at 28.8 FPS versus 11.5 FPS for Original NMS. Incorporating score penalty and distance ( $\text{Cluster-NMS}_{S+D}$ ) increases $\text{AR}_{100}$ to 43.8 (+3.5 over Fast NMS) and $\text{AR}_L$ to 66.4 (+5.6 over Fast NMS) while maintaining real-time inference (27.1 FPS).

Knowl 7 — Instance Segmentation Performance of CIoU and Cluster-NMS on BlendMask-RT

data/table

Instance segmentation performance on BlendMask-RT (ResNet-50-FPN) trained with Generalized-IoU (GIoU) loss versus Complete-IoU (CIoU) loss, evaluated across NMS variants on the MS COCO 2017 validation set on an NVIDIA GTX 1080Ti GPU:

Loss	NMS Strategy	FPS	Time (ms)	AP	$\text{AP}_{50}$	$\text{AP}_{75}$	$\text{AP}_S$	$\text{AP}_M$	$\text{AP}_L$	$\text{AR}_{100}$	$\text{AR}_L$
$\mathcal{L}_{GIoU}$	Fast NMS	42.7	23.4	34.4	54.3	36.3	14.4	38.3	51.9	46.8	63.4
$\mathcal{L}_{CIoU}$	Fast NMS	42.7	23.4	34.9	55.2	37.0	14.1	38.9	51.9	47.0	63.8
$\mathcal{L}_{CIoU}$	Original NMS (TorchVision)	42.7	23.4	35.1	55.4	37.2	14.2	39.1	52.2	48.5	65.9
$\mathcal{L}_{CIoU}$	Cluster-NMS	40.7	24.6	35.1	55.4	37.2	14.2	39.1	52.2	48.5	65.9
$\mathcal{L}_{CIoU}$	$\text{Cluster-NMS}_W$	39.7	25.2	35.2	55.4	37.3	14.3	39.2	52.5	48.7	66.2
$\mathcal{L}_{CIoU}$	$\text{Cluster-NMS}_{W+D}$	38.5	26.0	35.2	55.3	37.4	14.3	39.2	52.5	48.8	66.4

Replacing $\mathcal{L}_{GIoU}$ with $\mathcal{L}_{CIoU}$ improves mask AP from 34.4 to 34.9 under Fast NMS. Combining $\mathcal{L}_{CIoU}$ with $\text{Cluster-NMS}_{W+D}$ achieves 35.2 AP and 48.8 $\text{AR}_{100}$ (compared to 34.4 AP and 46.8 $\text{AR}_{100}$ for the $\mathcal{L}_{GIoU}$ baseline with Fast NMS) at 38.5 FPS.

Knowl 8 — Bounding Box Loss Comparison on YOLO v3 Object Detection

data/table

Object detection performance of YOLO v3 (Darknet608 backbone) trained on PASCAL VOC 07+12 trainval with various bounding box regression losses and evaluated on the PASCAL VOC 2007 test set using the GDarknet protocol and Original NMS:

Loss Function	AP (%)		$\text{AP}_{75}$ (%)
Loss Function	IoU metric	GIoU metric	IoU metric	GIoU metric
MSE	46.1	45.1	48.6	46.7
$\mathcal{L}_{IoU}$	46.6	45.8	49.8	48.8
$\mathcal{L}_{GIoU}$	47.7	46.9	52.2	51.1
$\mathcal{L}_{DIoU}$	48.1	47.4	52.8	51.9
$\mathcal{L}_{CIoU}$	49.2	48.4	54.3	52.9

Under the IoU evaluation metric, $\mathcal{L}_{CIoU}$ achieves 49.2% AP and 54.3% $\text{AP}_{75}$ , outperforming Mean-Squared Error (MSE) loss by +3.1% AP and +5.7% $\text{AP}_{75}$ , standard $\mathcal{L}_{IoU}$ by +2.6% AP and +4.5% $\text{AP}_{75}$ , $\mathcal{L}_{GIoU}$ by +1.5% AP and +2.1% $\text{AP}_{75}$ , and $\mathcal{L}_{DIoU}$ by +1.1% AP and +1.5% $\text{AP}_{75}$ . The same relative performance order holds under the GIoU metric.

Knowl 9 — Performance Comparison of NMS Variants on YOLO v3

data/table

Evaluation of NMS post-processing strategies using a pre-trained PyTorch YOLO v3 model (YOLOv3-spp-Ultralytics-608) on the MS COCO 2017 validation set:

NMS Strategy	FPS	Time (ms)	AP	$\text{AP}_{50}$	$\text{AP}_{75}$	$\text{AR}_1$	$\text{AR}_{10}$	$\text{AR}_{100}$	$\text{AR}_L$
Fast NMS	71.9	13.9	42.7	63.0	45.8	34.7	56.4	60.1	73.4
Original NMS (sequential)	9.6	103.9	43.2	63.2	46.5	34.7	57.7	62.7	75.8
Original NMS (TorchVision)	69.0	14.5	43.2	63.2	46.5	34.7	57.7	62.7	75.8
Weighted-NMS	6.2	162.3	43.6	63.4	47.4	34.9	58.1	63.0	75.5
Cluster-NMS	65.4	15.3	43.2	63.2	46.5	34.7	57.7	62.7	75.8
$\text{Cluster-NMS}_D$	60.6	16.5	43.3	63.0	47.1	34.7	58.1	63.6	76.7
$\text{Cluster-NMS}_W$	57.1	15.8	43.6	63.3	47.5	34.9	58.4	63.8	76.0
$\text{Cluster-NMS}_{W+D}$	53.5	17.1	43.8	63.0	47.9	34.9	58.8	64.6	77.1

The data shows that while sequential Weighted-NMS is computationally bottlenecked at 6.2 FPS (162.3 ms), $\text{Cluster-NMS}_W$ achieves the same AP (43.6) at 57.1 FPS (15.8 ms). Incorporating normalized distance in $\text{Cluster-NMS}_{W+D}$ achieves the highest overall accuracy (43.8 AP, 47.9 $\text{AP}_{75}$ ) and recall (64.6 $\text{AR}_{100}$ , 77.1 $\text{AR}_L$ ) at 53.5 FPS.

Knowl 10 — Evaluation of CIoU Loss and Cluster-NMS on SSD and Faster R-CNN

data/table

Quantitative comparison of bounding box regression losses and NMS methods on one-stage SSD (ResNet-50-FPN on PASCAL VOC 2007 test set) and two-stage Faster R-CNN (ResNet-50-FPN on MS COCO 2017 validation set):

SSD on PASCAL VOC 2007 (Loss Comparison with Original NMS)
Loss Function	AP	$\text{AP}_{75}$
$\mathcal{L}_{IoU}$	51.0	54.7
$\mathcal{L}_{GIoU}$	51.1	55.4
$\mathcal{L}_{DIoU}$	51.3	55.7
$\mathcal{L}_{CIoU}$	51.5	56.4
SSD on PASCAL VOC 2007 ( $\mathcal{L}_{CIoU}$ Model with NMS Methods)
NMS Strategy	FPS	Time (ms)	AP	$\text{AP}_{75}$
Fast NMS	28.8	34.7	50.7	56.2
Original NMS	17.8	56.1	51.5	56.4
Cluster-NMS	28.0	35.7	51.5	56.4
$\text{Cluster-NMS}_W$	26.8	37.3	51.9	56.3
$\text{Cluster-NMS}_{W+D}$	26.5	37.8	52.4	57.0
Faster R-CNN on MS COCO 2017 (Loss Comparison)
Loss Function	AP	$\text{AP}_{75}$	$\text{AP}_S$	$\text{AP}_M$	$\text{AP}_L$
$\mathcal{L}_{IoU}$ (Baseline)	37.9	40.8	21.6	40.8	50.1
$\mathcal{L}_{GIoU}$	38.0	41.1	21.5	41.1	50.2
$\mathcal{L}_{DIoU}$	38.1	41.1	21.7	41.2	50.3
$\mathcal{L}_{CIoU}$	38.7	42.0	21.3	41.9	51.5
$\mathcal{L}_{CIoU} + \text{Cluster-NMS}_{W+D}$	39.0	42.3	21.7	42.2	52.1

On SSD, $\mathcal{L}_{CIoU}$ achieves 51.5% AP and 56.4% $\text{AP}_{75}$ , and combining it with $\text{Cluster-NMS}_{W+D}$ increases AP to 52.4% at 26.5 FPS. On Faster R-CNN, $\mathcal{L}_{CIoU}$ improves AP from 37.9% to 38.7% over $\mathcal{L}_{IoU}$ , and applying $\text{Cluster-NMS}_{W+D}$ achieves 39.0% AP.

Knowl 11 — Scale-Dependent Impact of Aspect Ratio Penalty in CIoU Loss
limitation
While Complete-IoU (CIoU) loss improves overall Average Precision across one-stage, two-stage, and instance segmentation detectors, it yields minor performance drops on small or medium objects relative to Distance-IoU (DIoU) or IoU baselines (for instance, on MS COCO with Faster R-CNN, $\text{AP}_S$ is 21.3 for $\mathcal{L}_{CIoU}$ versus 21.7 for $\mathcal{L}_{DIoU}$ and 21.6 for $\mathcal{L}_{IoU}$ ; on YOLACT, $\text{AR}_S$ is 18.0 for $\mathcal{L}_{CIoU}$ versus 18.8 for $\text{Smooth-}\ell_1$ ).

This occurs because normalized central point distance is the primary geometric driver for small object localization, and enforcing aspect ratio consistency can weaken the gradient influence of normalized center distance. For large objects, both central distance and aspect ratio are crucial, producing consistent performance improvements (e.g., Faster R-CNN $\text{AP}_L$ increases from 50.1 to 51.5).

Coverage note — Table II (evaluating Cluster-NMS on original author-released YOLACT weights) was omitted as its conclusions are identical to Table I on the re-trained model.

References

1.X. Wang, M. Wang, and W. Li, “Scene-specific pedestrian detection for static video surveillance,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, no. 2, pp. 361–374, 2014.
2.P. Voigtlaender, J. Luiten, P. H. Torr, and B. Leibe, “Siam r-cnn: Visual tracking by re-detection,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
3.J. Marłn, D. Vzquez, A. M. Lpez, J. Amores, and L. I. Kuncheva, “rc-clusion handling via random subspace classifiers for human detection,” IEEE Transactions on Cybernetics, vol. 44, no. 3, pp. 342–354, 2014.
4.W. Wu, Y. Yin, X. Wang, and D. Xu, “Face detection with different scales based on faster r-cnn,” IEEE Transactions on Cybernetics, vol. 49, no. 11, pp. 4017–4028, 2019.
5.B. Xue and N. Tong, “rc-d: Fast and efficient weakly semi-supervised deep complex isar object detection,” IEEE Transactions on Cybernetics, vol. 49, no. 11, pp. 3991–4003, 2019.
6.J. Han, D. Zhang, G. Cheng, N. Liu, and D. Xu, “Advanced deep-learning techniques for salient and category-specific object detection: A survey,” IEEE Signal Processing Magazine, vol. 35, no. 1, pp. 84–100, 2018.
7.P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan, “Object detection with discriminatively trained part-based models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 9, pp. 1627–1645, 2009.
8.J. Redmon and A. Farhadi, “rc-9000: Better, faster, stronger,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 6517–6525.
9.J. Redmon and F. Ali, “rc-v3: An Incremental Improvement,” arXiv:1804.02767, 2018.
10.W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “rc- Single shot multibox detector,” in The European Conference on Computer Vision (ECCV). Springer, 2016, pp. 21–37.
11.C.-Y. Fu, W. Liu, A. Ranga, A. Tyagi, and A. C. Berg, “rc- Deconvolutional single shot detector,” arXiv:1701.06659, 2017.
12.T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollr, “rc-l loss for dense object detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 2, pp. 318–327, 2020.
13.Z. Tian, C. Shen, H. Chen, and T. He, “rc- Fully convolutional one-stage object detection,” in The IEEE International Conference on Computer Vision (ICCV), 2019, pp. 9626–9635.
14.R. Girshick, “rc- r-cnn,” in The IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1440–1448.
15.S. Ren, K. He, R. Girshick, and J. Sun, “rc-er r-cnn: Towards real-time object detection with region proposal networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1137–1149, 2017.
16.S. Gidaris and N. Komodakis, “rc-ct detection via a multi-region and semantic segmentation-aware cnn model,” in The IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1134–1142.
17.Z. Cai and N. Vasconcelos, “rc-cade r-cnn: Delving into high quality object detection,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 6154–6162.
18.X. Chen, R. Girshick, K. He, and P. Dollar, “rc-sormask: A foundation for dense object segmentation,” in The IEEE International Conference on Computer Vision (ICCV), 2019, pp. 2061–2069.
19.E. Xie, P. Sun, X. Song, W. Wang, X. Liu, D. Liang, C. Shen, and P. Luo, “rc-armask: Single shot instance segmentation with polar representation,” arXiv preprint arXiv:1909.13226, 2019.
20.D. Bolya, C. Zhou, F. Xiao, and Y. J. Lee, “rc-act: real-time instance segmentation,” in The IEEE International Conference on Computer Vision (ICCV), 2019, pp. 9157–9166.
21.K. He, G. Gkioxari, P. Dollr, and R. Girshick, “rc-k r-cnn,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 2, pp. 386–397, 2020.
22.P. J. Huber et al., “rc-ust estimation of a location parameter,” The Annals of Mathematical Statistics, vol. 35, no. 1, pp. 73–101, 1964.
23.S.-H. Bae, “rc-ct detection based on region decomposition and assembly,” in The AAAI Conference on Artificial Intelligence, vol. 33, 2019, pp. 8094–8101.
24.G. Cheng, J. Han, P. Zhou, and D. Xu, “rc-rning rotation-invariant and fisher discriminative convolutional neural networks for object detection,” IEEE Transactions on Image Processing, vol. 28, no. 1, pp. 265–278, 2019.
25.G. Brazil, X. Yin, and X. Liu, “rc-uminating pedestrians via simultaneous detection & segmentation,” in The IEEE International Conference on Computer Vision (ICCV), 2017, pp. 4960–4969.
26.C. Zhou, M. Wu, and S.-K. Lam, “rc-a-cnn: Semantic self-attention cnn for pedestrian detection,” arXiv preprint arXiv:1902.09080, 2019.
27.J. Ma, W. Shao, H. Ye, L. Wang, H. Wang, Y. Zheng, and X. Xue, “rc-itrary-oriented scene text detection via rotation proposals,” IEEE Transactions on Multimedia, vol. 20, no. 11, pp. 3111–3122, 2018.
28.M. Liao, Z. Zhu, B. Shi, G.-s. Xia, and X. Bai, “rc-ation-sensitive regression for oriented scene text detection,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 5909–5918.
29.S. Qin, A. Bissacco, M. Raptis, Y. Fujii, and Y. Xiao, “rc-ards unconstrained end-to-end text spotting,” in The IEEE International Conference on Computer Vision (ICCV), 2019, pp. 4704–4714.
30.Y. Zhou and O. Tuzel, “rc-elnet: End-to-end learning for point cloud based 3d object detection,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 4490–4499.
31.S. Shi, X. Wang, and H. Li, “rc-ntrcnn: 3d object proposal generation and detection from point cloud,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 770–779.
32.K. Sun, B. Xiao, D. Liu, and J. Wang, “rc-p high-resolution representation learning for human pose estimation,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 5693–5703.
33.K. Iskakov, E. Burkov, V. Lempitsky, and Y. Malkov, “rc-rnable triangulation of human pose,” in The IEEE International Conference on Computer Vision (ICCV), 2019, pp. 7718–7727.
34.J. Yu, Y. Jiang, Z. Wang, Z. Cao, and T. Huang, “rc-tbox: An advanced object detection network,” in Proceedings of the ACM International Conference on Multimedia, 2016, pp. 516–520.
35.L. Tychsen-Smith and L. Petersson, “rc-roving object localization with fitness nms and bounded iou loss,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 6877–6885.
36.H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese, “Generalized intersection over union: A metric and a loss for bounding box regression,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 658–666.
37.R. Girshick, J. Donahue, T. Darrell, and J. Malik, “rc-h feature hierarchies for accurate object detection and semantic segmentation,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 580–587.
38.J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “rc- only look once: Unified, real-time object detection,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779–788.
39.N. Bodla, B. Singh, R. Chellappa, and L. S. Davis, “Soft-nms ł improving object detection with one line of code,” in The IEEE International Conference on Computer Vision (ICCV), 2017, pp. 5562–5570.
40.H. Zhou, Z. Li, C. Ning, and J. Tang, “rc- Scale invariant framework for real-time object detection,” in The IEEE International Conference on Computer Vision (ICCV Workshop), 10 2017, pp. 760–768.
41.D. Oro, C. Fernandez, X. Martorell, and J. Hernando, “rc-k-efficient parallel non-maximum suppression for embedded gpu architectures,” in The IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016, pp. 1026–1030.
42.H. Chen, K. Sun, Z. Tian, C. Shen, Y. Huang, and Y. Yan, “BlendMask: Top-down meets bottom-up for instance segmentation,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
43.Z. Zheng, P. Wang, W. Liu, J. Li, R. Ye, and D. Ren, “rc-tance-IoU Loss: Faster and better learning for bounding box regression,” in The AAAI Conference on Artificial Intelligence, 2020.
44.P. Zhou, B. Ni, C. Geng, J. Hu, and Y. Xu, “rc-le-transferrable object detection,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 528–537.
45.Z. Yang, S. Liu, H. Hu, L. Wang, and S. Lin, “rc-points: Point set representation for object detection,” in The IEEE International Conference on Computer Vision (ICCV), 2019, pp. 9656–9665.
46.Y. Jiang, X. Zhu, X. Wang, S. Yang, W. Li, H. Wang, P. Fu, and Z. Luo, “rc-nn: Rotational region cnn for orientation robust scene text detection,” arXiv preprint arXiv:1706.09579, 2017.
47.G. P. Meyer, “An alternative probabilistic interpretation of the huber loss,” arXiv preprint arXiv:1911.02088, 2019.
48.J. Pang, K. Chen, J. Shi, H. Feng, W. Ouyang, and D. Lin, “rc-ra r-cnn: Towards balanced learning for object detection,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 821–830.
49.B. Li, Y. Liu, and X. Wang, “rc-adient harmonized single-stage detector,” in The AAAI Conference on Artificial Intelligence, vol. 33, 2019, pp. 8577–8584.
50.B. Jiang, R. Luo, J. Mao, T. Xiao, and Y. Jiang, “rc-uisition of localization confidence for accurate object detection,” in The European Conference on Computer Vision (ECCV), 2018, pp. 784–799.
51.S. Liu, D. Huang, and Y. Wang, “rc-aptive nms: Refining pedestrian detection in a crowd,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 6452–6461.
52.Y. He, C. Zhu, J. Wang, M. Savvides, and X. Zhang, “rc-unding box regression with uncertainty for accurate object detection,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 2883–2892.
53.M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” International Journal of Computer Vision, vol. 88, no. 2, pp. 303–338, 2010.
54.T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick, “rc-rosoft coco: Common objects in context,” in The European Conference on Computer Vision (ECCV). Springer, 2014, pp. 740–755.

Citation

MLA

Zheng, Z., et al. “Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentation”. arXiv, 2020, http://arxiv.org/abs/2005.03572v4.

APA

Zheng, Z., Wang, P., Ren, D., Liu, W., Ye, R., Hu, Q., & Zuo, W. (2020). Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentation. arXiv. http://arxiv.org/abs/2005.03572v4

Chicago

Zheng, Z., P. Wang, D. Ren, et al. 2020. “Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentation”. arXiv. http://arxiv.org/abs/2005.03572v4.

Harvard

Zheng, Z. et al. (2020) “Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentation”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2005.03572v4.

Vancouver

1. Zheng Z, Wang P, Ren D, Liu W, Ye R, Hu Q, Zuo W (2020) Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentation. arXiv

BibTeX

@article{zheng2020enhancing,
  title = {Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentation},
  author = {Zheng, Zhaohui and Wang, Ping and Ren, Dongwei and Liu, Wei and Ye, Rongguang and Hu, Qinghua and Zuo, Wangmeng},
  year = {2020},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2005.03572v4},
  eprint = {2005.03572}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentation

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Complete-IoU (CIoU) Loss Formulation

Knowl 2 — Cluster-NMS Algorithm

Knowl 3 — Equivalence and Convergence Bound of Cluster-NMS to Sequential NMS

Knowl 4 — Geometric Extensions of Cluster-NMS

Knowl 5 — Simulation Experiment Protocol for Evaluating Bounding Box Regression Losses

Knowl 6 — Instance Segmentation Performance of CIoU and Cluster-NMS on YOLACT

Knowl 7 — Instance Segmentation Performance of CIoU and Cluster-NMS on BlendMask-RT

Knowl 8 — Bounding Box Loss Comparison on YOLO v3 Object Detection

Knowl 9 — Performance Comparison of NMS Variants on YOLO v3

Knowl 10 — Evaluation of CIoU Loss and Cluster-NMS on SSD and Faster R-CNN

Knowl 11 — Scale-Dependent Impact of Aspect Ratio Penalty in CIoU Loss

References

Citation

BibTeX

Source Code

Access the Paper