YOLACT: Real-Time Instance Segmentation

Daniel BolyaChong ZhouFanyi XiaoYong Jae Lee

article2019ICCV2,186 citations

Introduces YOLACT, a fully convolutional framework that achieves the first real-time instance segmentation at over 30 frames per second on MS COCO by decomposing mask generation into parallel prototype and coefficient prediction subtasks.

Listen

Modern computer vision applications increasingly require identifying individual objects and outlining their exact pixel boundaries, a task known as instance segmentation. While standard object detection models can operate at real-time speeds (above 30 frames per second), existing instance segmentation systems rely on complex, multi-stage architectures that crop and re-examine localized image regions. This sequential processing creates a severe computational bottleneck, preventing high-accuracy instance segmentation from operating in real-time settings like video analytics and autonomous systems.

The article introduces and evaluates YOLACT (You Only Look At CoefﬁcienTs), a simple, single-stage framework designed to achieve real-time instance segmentation without sacrificing boundary quality. The researchers sought to demonstrate that separating the task into parallel components can eliminate the speed penalty of sequential region extraction while delivering competitive accuracy on standard vision benchmarks.

To evaluate this approach, the authors designed a fully-convolutional model trained on standard industry datasets, primarily Microsoft Common Objects in Context (MS COCO) and Pascal VOC. Rather than repooling localized features sequentially, the framework splits instance segmentation into two parallel subtasks: generating a dictionary of non-local "prototype masks" across the entire image and predicting per-instance "mask coefficients." Final segmentations are assembled using lightweight matrix multiplication to combine prototypes linearly before cropping them with predicted bounding boxes. The authors also developed "Fast NMS," an accelerated parallel technique for filtering duplicate detections, and evaluated various backbone networks across different image resolutions on a single graphics processing unit (GPU).

The evaluation produced several key findings. First, the base YOLACT configuration achieved a speed of 33.5 frames per second on MS COCO with a 29.8 mean Average Precision (mAP), making it nearly four times faster than previous competitive models and the first to achieve real-time performance at this accuracy level. Second, the mask assembly process adds only about 5 to 6 milliseconds of overhead to the underlying detector, demonstrating extreme computational efficiency. Third, the system produces higher-quality, sharper masks for large objects and achieves superior temporal stability across video frames compared to standard multi-stage approaches, outperforming them at strict overlap thresholds (1.6 versus 1.3 mAP at 95% intersection-over-union). Fourth, the Fast NMS technique yielded an approximate 12-millisecond speedup with only a negligible 0.1 mAP reduction in accuracy. Finally, using 32 prototype masks proved optimal, as smaller vocabularies caused underfitting and larger ones increased coefficient complexity without meaningful gains.

These results indicate that real-time instance segmentation is commercially and technically viable on consumer-grade hardware without costly multi-GPU infrastructure or heavy post-processing pipelines. The temporal stability and crisp boundary delineation make the method particularly compelling for live video streams and latency-sensitive deployments. Although YOLACT trails top-tier two-stage baselines by roughly 6 points in overall mask mAP, the analysis reveals that this performance gap stems from bounding-box detection inaccuracies rather than the mask generation mechanism itself.

Organizations aiming to deploy instance segmentation in latency-critical environments should consider adopting single-stage parallel architectures like YOLACT. To maximize throughput, teams can use Fast NMS and lighter backbones (such as ResNet-50 or DarkNet-53) rather than reducing input resolution, which severely harms mask quality. Future engineering and research efforts should focus on combining YOLACT’s mask branch with more advanced one-stage object detectors (incorporating multi-scale training and anchor optimization) to close the remaining detection gap.

Decision-makers should note certain operational boundaries: the model struggles with "localization failure" in highly crowded scenes with overlapping objects and can exhibit mask "leakage" when predicted bounding boxes are oversized or misaligned. Nevertheless, the findings provide strong confidence that parallel prototype-coefficient architectures offer an effective, production-ready foundation for real-time computer vision tasks.

University of California, Davis gpu marginals network management systems real-time instance segmentation real-time object detection semantic segmentation

arXiv: 1904.02689 dbolya/yolact

Paper: Mask R-CNN, Kaiming He et al. (2017). Reading Mask R-CNN provides essential context on two-stage instance segmentation baselines that directly motivated YOLOC's single-stage design.
Paper: Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, Shaoqing Ren et al. (2015). Understanding Faster R-CNN is crucial because YOLACT builds directly upon region-proposal and anchor concepts to perform real-time instance segmentation.
Paper: SSD: Single Shot MultiBox Detector, W. Liu et al. (2015). SSD establishes the foundational single-shot object detection principles that inform YOLACT's fully-convolutional, real-time approach.

Paper: YOLOv4: Optimal Speed and Accuracy of Object Detection, Alexey Bochkovskiy et al. (2020). YOLOv4 extends real-time detection architectures and builds directly upon the efficiency advances pioneered by models like YOLACT.
Paper: YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors, Chien-Yao Wang et al. (2023). YOLOv7 continues the push for real-time visual efficiency, generalizing the speed-accuracy tradeoffs established in YOLACT.
Paper: YOLOv10: Real-Time End-to-End Object Detection, Ao Wang et al. (2024). YOLOv10 extends the real-time detection paradigm further by eliminating post-processing bottlenecks like NMS discussed in YOLACT.

Cover for YOLACT: Real-Time Instance Segmentation

Abstract

We present a simple, fully-convolutional model for real-time instance segmentation that achieves 29.8 mAP on MS COCO at 33.5 fps evaluated on a single Titan Xp, which is significantly faster than any previous competitive approach. Moreover, we obtain this result after training on only one GPU. We accomplish this by breaking instance segmentation into two parallel subtasks: (1) generating a set of prototype masks and (2) predicting per-instance mask coefficients. Then we produce instance masks by linearly combining the prototypes with the mask coefficients. We find that because this process doesn't depend on repooling, this approach produces very high-quality masks and exhibits temporal stability for free. Furthermore, we analyze the emergent behavior of our prototypes and show they learn to localize instances on their own in a translation variant manner, despite being fully-convolutional. Finally, we also propose Fast NMS, a drop-in 12 ms faster replacement for standard NMS that only has a marginal performance penalty.

Topics

gpu marginals network management systems

Keywords

real-time instance segmentation real-time object detection semantic segmentation instance masks feature pyramid COCO test-dev segmentation loss feature repooling

1 Introduction
2 Related Work
3 YOLACT
3.1 Prototype Generation
3.2 Mask Coefficients
3.3 Mask Assembly
3.4 Emergent Behavior
4 Backbone Detector
5 Other Improvements
6 Results
7 Discussion
A Appendix
A.1 Box Results
A.2 More Qualitative Results
References

Knowls

Knowl 1 — YOLACT Dual-Branch Instance Segmentation Paradigm
model/method
YOLACT (You Only Look At CoefficienTs) is a one-stage, fully-convolutional real-time instance segmentation framework that eliminates explicit region localization and feature repooling (such as RoIPool or RoIAlign). The model divides instance segmentation into two parallel subtasks:
1. Prototype Mask Generation: A fully convolutional network branch (termed Protonet) processes backbone features to predict a set of $k$ non-local, image-sized prototype masks for the entire image, independent of any individual instance.
2. Mask Coefficient Prediction: An extra head attached in parallel to the bounding box and classification branches of an anchor-based object detector predicts a vector of $k$ linear combination coefficients per anchor, representing the instance's representation in the prototype space.
Final instance masks are synthesized via a GPU-accelerated matrix multiplication that linearly combines the predicted prototype masks using the instance's predicted mask coefficients, followed by a sigmoid nonlinearity and spatial cropping using the predicted bounding box. Because prototype generation and coefficient prediction are computed independently in parallel, the mask generation branch adds only $\sim 5\text{ ms}$ of computational overhead to the base detector.
Knowl 2 — Protonet Architecture for Instance Mask Prototypes
model/method
The prototype generation branch (Protonet) is a fully convolutional network that outputs a dictionary of $k$ prototype masks of size $H_p \times W_p$ for an input image of dimensions $H \times W$ (where $H_p = H/4$ and $W_p = W/4$ ).

Protonet is attached to the deepest and largest feature layer of a Feature Pyramid Network (FPN), specifically the $P_3$ feature map ( $69 \times 69 \times 256$ for a $550 \times 550$ input image). The architecture consists of:
1. Three successive $3 \times 3$ convolutional layers with 256 channels and ReLU activations.
2. An upsampling step (bilinear interpolation) to double spatial resolution followed by a $3 \times 3$ convolution (256 channels, ReLU).
3. A final $1 \times 1$ convolution with $k$ output channels (where $k=32$ by default).
4. An unbounded or ReLU activation at the output to produce interpretable prototypes and allow strong activations on confident regions.
Protonet receives no direct prototype-level supervision or auxiliary prototype target loss; all supervision flows backward from the post-assembly binary cross-entropy mask loss.
Knowl 3 — Mask Assembly Formulation and Cropping
equation
For an image containing $n$ detected object instances that survive non-maximum suppression and score thresholding, final instance masks $M$ are produced by matrix multiplication of the prototype masks and the mask coefficient matrix:

$M = \sigma(P C^T)$

where:
- $P \in \mathbb{R}^{h \times w \times k}$ is the tensor of $k$ prototype masks generated by Protonet, reshaped to an $(h \cdot w) \times k$ matrix where $h = H/4$ and $w = W/4$ .
- $C \in \mathbb{R}^{n \times k}$ is the matrix of predicted mask coefficients for the $n$ instances, where each row represents the $k$ -dimensional coefficient vector predicted for an instance anchor, activated with a $\tanh$ nonlinearity to allow subtraction of prototype features.
- $\sigma(x) = \frac{1}{1 + e^{-x}}$ is the element-wise sigmoid function applied to the linear combination.
- $M \in \mathbb{R}^{n \times h \times w}$ is the resulting tensor of $n$ assembled instance masks.
Each assembled mask $M_i$ ( $i \in \{1, \dots, n\}$ ) is subsequently cropped by zeroing all pixels outside its corresponding bounding box. During training, masks are cropped using ground-truth bounding boxes; during inference, masks are cropped using predicted bounding boxes evaluated at prototype resolution ( $h \times w$ ) with 1-pixel padding.
Knowl 4 — Fast Non-Maximum Suppression (Fast NMS)
algorithm
Fast NMS replaces sequential class-wise Non-Maximum Suppression with parallelized GPU tensor operations by allowing detections that have already been flagged for removal to also suppress other detections. This relaxation enables fully vectorized execution across all $c$ classes and top $n$ detections per class.
Input: Tensor of detected bounding boxes and scores for $c$ classes, sorted in descending order of score, top $n$ per class; IoU threshold $t$
Output: Boolean keep mask $K_{\text{keep}} \in \{0, 1\}^{c \times n}$
Compute pairwise IoU tensor $X \in \mathbb{R}^{c \times n \times n}$ where $X_{kij} = \text{IoU}(\text{box}_{ki}, \text{box}_{kj})$ for class $k$
Zero out lower triangle and diagonal: $X_{kij} \leftarrow 0$ for all $k \in \{1, \dots, c\}$ and all $i \ge j$
Compute maximum column-wise IoU: $K_{kj} \leftarrow \max_i (X_{kij})$ for all $k \in \{1, \dots, c\}, j \in \{1, \dots, n\}$
Determine keep status by thresholding: $K_{\text{keep}, kj} \leftarrow (K_{kj} < t)$
return $K_{\text{keep}}$
While Fast NMS can discard slightly more overlapping true positive boxes than standard NMS, it executes in $\sim 1.4\text{ ms}$ instead of $\sim 13.2\text{ ms}$ on a Titan Xp GPU, saving 11.8 ms to 15.0 ms with a marginal accuracy penalty of $0.1$ to $0.3$ mask mAP.
Knowl 5 — YOLACT Multi-Task Loss and Auxiliary Semantic Head
model/method
The YOLACT network is trained end-to-end using a multi-task loss function combining classification, bounding box regression, assembled mask segmentation, and an auxiliary semantic segmentation loss:

$L = L_{\text{cls}} + 1.5 L_{\text{box}} + 6.125 L_{\text{mask}} + L_{\text{seg}}$
1. Classification Loss ( $L_{\text{cls}}$ ): Softmax cross-entropy over $c$ foreground classes and 1 background class, using Online Hard Example Mining (OHEM) with a 3:1 negative-to-positive ratio.
2. Bounding Box Loss ( $L_{\text{box}}$ ): Smooth- $L_1$ loss over SSD-style coordinate regression offsets.
3. Mask Loss ( $L_{\text{mask}}$ ): Pixel-wise binary cross-entropy (BCE) between assembled masks $M$ cropped to ground-truth boxes and binary ground-truth masks $M_{\text{gt}}$ , normalized by the ground-truth bounding box area $A_{\text{gt}}$ :
$L_{\text{mask}} = \frac{1}{A_{\text{gt}}} \text{BCE}(M_{\text{crop}}, M_{\text{gt}})$
1. Auxiliary Semantic Segmentation Loss ( $L_{\text{seg}}$ ): A training-only $1 \times 1$ convolutional layer with $c$ output channels attached directly to the $P_3$ FPN feature layer. It predicts per-pixel class presence supervised via sigmoid cross-entropy against pixel-level class assignments constructed from instance ground-truth. Evaluated only during training, it increases feature richness and delivers a $+0.4\text{ mAP}$ boost without any test-time computational penalty.
Knowl 6 — Emergent Translation Variance in Fully Convolutional Prototypes
theoretical result
Although strictly convolutional operations without padding are translation-invariant (yielding identical outputs for shifted identical inputs), modern FCN backbones (such as ResNet) introduce translation variance through zero-padding applied at feature borders. Stacked convolutional layers propagate boundary zero-padding inward, providing the network with implicit coordinates relative to image edges.

Consequently, the prototype generation branch (Protonet) learns translation-variant spatial functions without explicit coordinate channels. The learned prototypes exhibit distinct emergent functional behaviors:
1. Soft Spatial Partitioning: Prototypes that activate only on one side of an implicit learned image boundary (e.g., left half, bottom-left quadrant), enabling the linear combination to separate adjacent or overlapping instances of identical semantic classes by subtracting partition prototypes.
2. Position-Sensitive Directional Maps: Prototypes that fire selectively on specific sub-regions of objects (such as the bottom-left edges of all instances).
3. Boundary and Background Detectors: Prototypes that identify contours, inter-object silhouettes, and global background/ground planes.
Because individual prototypes compress multiple behaviors, linear combination with $k=32$ prototypes suffices to uniquely localize and separate multiple overlapping instances across an entire image.

Knowl 7 — MS COCO Instance Segmentation Benchmark Performance

data/table

The following table compares YOLACT model variants with state-of-the-art instance segmentation approaches on the MS COCO test-dev benchmark. All inference speeds are measured on a single NVIDIA Titan Xp GPU without test-time augmentations.

Method	Backbone	FPS	Time (ms)	$\text{AP}$	$\text{AP}_{50}$	$\text{AP}_{75}$	$\text{AP}_{S}$	$\text{AP}_{M}$	$\text{AP}_{L}$
PA-Net	R-50-FPN	4.7	212.8	36.6	58.0	39.3	16.3	38.1	53.1
RetinaMask	R-101-FPN	6.0	166.7	34.7	55.4	36.9	14.3	36.7	50.5
FCIS	R-101-C5	6.6	151.5	29.5	51.5	30.2	8.0	31.0	49.7
Mask R-CNN	R-101-FPN	8.6	116.3	35.7	58.0	37.8	15.5	38.1	52.4
MS R-CNN	R-101-FPN	8.6	116.3	38.3	58.8	41.5	17.8	40.4	54.4
YOLACT-550	R-101-FPN	33.5	29.8	29.8	48.5	31.2	9.9	31.3	47.7
YOLACT-400	R-101-FPN	45.3	22.1	24.9	42.0	25.4	5.0	25.3	45.0
YOLACT-550	R-50-FPN	45.0	22.2	28.2	46.6	29.2	9.2	29.3	44.8
YOLACT-550	D-53-FPN	40.7	24.6	28.7	46.8	30.0	9.5	29.6	45.5
YOLACT-700	R-101-FPN	23.4	42.7	31.2	50.6	32.8	12.1	33.3	47.1

YOLACT-550 with ResNet-101 is the first model to surpass real-time thresholds ( $>30\text{ FPS}$ ) on MS COCO while maintaining competitive accuracy ( $29.8\text{ mAP}$ at $33.5\text{ FPS}$ ), operating $3.9\times$ faster than the prior fastest competitive approach (Mask R-CNN at $8.6\text{ FPS}$ ).

Knowl 8 — Ablation Analysis of Fast NMS and Prototype Count
empirical result
Ablation experiments evaluated on MS COCO val2017 establish the trade-offs of Fast NMS and the number of prototype masks $k$ :
1. Fast NMS vs. Standard NMS:
  - In YOLACT-550, Standard NMS achieves $30.0\text{ mask mAP}$ at $24.0\text{ FPS}$ ( $41.6\text{ ms}$ latency), while Fast NMS achieves $29.9\text{ mask mAP}$ at $33.5\text{ FPS}$ ( $29.8\text{ ms}$ latency), yielding an $11.8\text{ ms}$ speedup for a $0.1\text{ mAP}$ drop.
  - In Mask R-CNN, Fast NMS reduces latency from $116.0\text{ ms}$ ( $8.6\text{ FPS}$ ) to $101.0\text{ ms}$ ( $9.9\text{ FPS}$ ), an improvement of $15.0\text{ ms}$ with a $0.3\text{ mAP}$ drop ( $36.1$ vs. $35.8$ ).
2. Number of Prototypes ( $k$ ):
  - Models evaluated after 400k iterations show $k=8$ achieves $26.8\text{ mAP}$ at $33.0\text{ FPS}$ ( $30.4\text{ ms}$ ); $k=16$ achieves $27.1\text{ mAP}$ at $32.8\text{ FPS}$ ( $30.5\text{ ms}$ ); $k=32$ achieves $27.7\text{ mAP}$ at $32.4\text{ FPS}$ ( $30.9\text{ ms}$ ); $k=64$ achieves $27.8\text{ mAP}$ at $31.7\text{ FPS}$ ( $31.5\text{ ms}$ ); $k=128$ achieves $27.6\text{ mAP}$ at $31.5\text{ FPS}$ ( $31.8\text{ ms}$ ); and $k=256$ achieves $27.7\text{ mAP}$ at $29.8\text{ FPS}$ ( $33.6\text{ ms}$ ).
  - Increasing $k$ beyond 32 provides negligible performance gains because predicting higher-dimensional coefficient vectors becomes harder to optimize, causing redundant prototype channels with small edge variations.
Knowl 9 — Mask Boundary Fidelity and Video Temporal Stability
empirical result
Because YOLACT generates instance masks at $138 \times 138$ resolution directly from full-image feature representations without intermediate feature cropping or repooling (e.g., RoIPool or RoIAlign), it avoids feature distortion and spatial misalignment on large objects:
1. High-Overlap Metric ( $AP_{95}$ ): On MS COCO test-dev, YOLACT-550 achieves $1.6\text{ AP}$ at the strict $95\%$ IoU threshold, outperforming Mask R-CNN ( $1.3\text{ AP}$ ) despite Mask R-CNN having higher overall mAP ( $35.7$ vs. $29.8$ ). This demonstrates superior boundary accuracy on large instances when localization is correct.
2. Video Temporal Stability: When evaluated on sequential video frames without explicit temporal smoothing, YOLACT generates temporally consistent masks across frames. In two-stage methods, masks jitter between frames because RoIAlign features depend directly on slight frame-to-frame variations in first-stage proposal boxes. In YOLACT, the underlying prototype masks are computed globally across the entire frame, preventing box jitter from degrading the mask representation.
Knowl 10 — YOLACT Failure Modes: Mask Leakage, Localization Failure, and Detector Error
limitation
YOLACT exhibits two specific mask-generation failure modes along with a detector-level accuracy bottleneck:
1. Localization Failure: When multiple instances of the same class densely cluster in a single image region, Protonet can fail to allocate distinct prototype activations to separate each instance. In such cases, the assembled mask outputs a single composite foreground mask covering the cluster rather than individual instance segmentations.
2. Mask Leakage: Because YOLACT does not suppress activations outside the instance region in the prototype maps (relying on post-assembly bounding box cropping), an oversized or misaligned bounding box causes background noise or masks of distant instances to leak into the instance prediction.
3. Detector Performance Bottleneck: On MS COCO test-dev, YOLACT-550 has a mask mAP of $29.8$ and a bounding box mAP of $32.3$ (a difference of $2.5\text{ mAP}$ ). Mask R-CNN exhibits an identical $2.5\text{ mAP}$ gap ( $35.7\text{ mask mAP}$ vs. $38.2\text{ box mAP}$ ). This demonstrates that the overall accuracy gap between YOLACT and two-stage methods is primarily driven by errors in the one-stage detector (misclassifications and box localization errors) rather than flaws in the linear mask assembly mechanism.

Coverage note — Pascal 2012 SBD benchmark results and standalone YOLOv3 object detection comparisons were omitted as secondary validations that do not alter the core algorithmic or empirical findings of the paper.

References

1.Shivani Agarwal and Dan Roth. Learning a sparse representation for object detection. In ECCV, 2002.
2.Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. CoRR, 2015.
3.Min Bai and Raquel Urtasun. Deep watershed transform for instance segmentation. In CVPR, 2017.
4.Liang-Chieh Chen, Alexander Hermans, George Papandreou, Florian Schroff, Peng Wang, and Hartwig Adam. Masklab: Instance segmentation by refining object detection with semantic and direction features. In CVPR, 2018.
5.Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
6.Jifeng Dai, Kaiming He, Yi Li, Shaoqing Ren, and Jian Sun. Instance-sensitive fully convolutional networks. In ECCV, 2016.
7.Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware semantic segmentation via multi-task network cascades. In CVPR, 2016.
8.Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-fcn: Object detection via region-based fully convolutional networks. In NeurIPS, 2016.
9.Bert De Brabandere, Davy Neven, and Luc Van Gool. Semantic instance segmentation with a discriminative loss function. arXiv preprint arXiv:1708.02551, 2017.
10.Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A Large-Scale Hierarchical Image Database. In CVPR, 2009.
11.Nikita Dvornik, Konstantin Shmelkov, Julien Mairal, and Cordelia Schmid. Blitznet: A real-time deep network for scene understanding. In ICCV, 2017.
12.Mark Everingham, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303–338, June 2010.
13.Alireza Fathi, Zbigniew Wojna, Vivek Rathod, Peng Wang, Hyun Oh Song, Sergio Guadarrama, and Kevin Murphy. Semantic instance segmentation via deep metric learning. arXiv preprint arXiv:1703.10277, 2017.
14.Cheng-Yang Fu, Mykhailo Shvets, and Alexander C Berg. Retinamask: Learning to predict masks improves state-of-the-art single-shot detection for free. arXiv preprint arXiv:1901.03353, 2019.
15.Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
16.Bharath Hariharan, Pablo Arbelaez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In ICCV, 2011.
17.Adam Harley, Konstantinos Derpanis, and Iasonas Kokkinos. Segmentation-aware convolutional networks using local attention masks. In ICCV, 2017.
18.Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Gir- shick. Mask r-cnn. In ICCV, 2017.
19.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
20.Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang Huang, and Xinggang Wang. Mask scoring r-cnn. In CVPR, 2019.
21.Saumya Jetley, Michael Sapienza, Stuart Golodetz, and Philip Torr. Straight to shapes: real-time detection of encoded shapes. In CVPR, 2017.
22.Alexander Kirillov, Evgeny Levinkov, Bjoern Andres, Bogdan Savchynskyy, and Carsten Rother. Instancecut: from edges to instances with multicut. In CVPR, 2017.
23.Thomas Leung and Jitendra Malik. Representing and recognizing the visual appearance of materials using three-dimensional textons. IJCV, 2001.
24.Yi Li, Haozhi Qi, Jifeng Dai, Xiangyang Ji, and Yichen Wei. Fully convolutional instance-aware semantic segmentation. In CVPR, 2017.
25.Xiaodan Liang, Liang Lin, Yunchao Wei, Xiaohui Shen, Jianchao Yang, and Shuicheng Yan. Proposal-free network for instance-level object segmentation. TPAMI, 2018.
26.Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
27.Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. In CVPR, 2017.
28.Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
29.Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmentation. In CVPR, 2018.
30.Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander Berg. Ssd: Single shot multibox detector. In ECCV, 2016.
31.Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
32.Alejandro Newell, Zhiao Huang, and Jia Deng. Associative embedding: End-to-end learning for joint detection and grouping. In NeurIPS, 2017.
33.Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and Eugenio Culurciello. Enet: A deep neural network architecture for real-time semantic segmentation. CoRR, 2016.
34.Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016.
35.Joseph Redmon and Ali Farhadi. Yolo9000: Better, faster, stronger. In CVPR, 2017.
36.Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv:1804.02767, 2018.
37.Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.
38.Xiaofeng Ren and Deva Ramanan. Histograms of sparse codes for object detection. In CVPR, 2013.
39.Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In CVPR, 2016.
40.Josef Sivic and Andrew Zisserman. Video google: A text retrieval approach to object matching in videos. In ICCV, 2003.
41.Michael Treml, Jose Arjona-Medina, Thomas Unterthiner, Rupesh Durgesh, Felix Friedmann, Peter Schuberth, Andreas Mayr, Martin Heusel, Markus Hofmarcher, Michael Widrich, et al. Speeding up semantic segmentation for autonomous driving. In NeurIPS Workshops, 2016.
42.Jonas Uhrig, Eike Rehder, Bjorn Fr ohlich, Uwe Franke, and Thomas Brox. Box2pix: Single-shot instance segmentation by assigning pixels to object boxes. In IEEE Intelligent Vehicles Symposium, 2018.
43.Jinjun Wang, Jianchao Yang, Kai Yu, Fengjun Lv, Thomas Huang, and Yihong Gong. Locality-constrained linear coding for image classification. In CVPR, 2010.
44.Jianchao Yang, John Wright, Thomas Huang, and Yi Ma. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 2010.
45.Xiaodong Yu, Li Yi, Cornelia Fermuller, and David Doer- mann. Object detection using shape codebook. In BMVC, 2007.
46.Tianzhu Zhang, Bernard Ghanem, Si Liu, Changsheng Xu, and Narendra Ahuja. Low-rank sparse coding for image classification. In ICCV, 2013.
47.Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, and Jiaya Jia. Icnet for real-time semantic segmentation on high-resolution images. In ECCV, 2018.

Citation

MLA

Bolya, D., et al. “YOLACT: Real-time Instance Segmentation”. arXiv, 2019, http://arxiv.org/abs/1904.02689v2.

APA

Bolya, D., Zhou, C., Xiao, F., & Lee, Y. J. (2019). YOLACT: Real-time Instance Segmentation. arXiv. http://arxiv.org/abs/1904.02689v2

Chicago

Bolya, D., C. Zhou, F. Xiao, and Y. J. Lee. 2019. “YOLACT: Real-time Instance Segmentation”. arXiv. http://arxiv.org/abs/1904.02689v2.

Harvard

Bolya, D. et al. (2019) “YOLACT: Real-time Instance Segmentation”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1904.02689v2.

Vancouver

1. Bolya D, Zhou C, Xiao F, Lee YJ (2019) YOLACT: Real-time Instance Segmentation. arXiv

BibTeX

@article{bolya2019yolact,
  title = {YOLACT: Real-time Instance Segmentation},
  author = {Bolya, Daniel and Zhou, Chong and Xiao, Fanyi and Lee, Yong Jae},
  year = {2019},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1904.02689v2},
  eprint = {1904.02689}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE

YOLACT: Real-Time Instance Segmentation

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — YOLACT Dual-Branch Instance Segmentation Paradigm

Knowl 2 — Protonet Architecture for Instance Mask Prototypes

Knowl 3 — Mask Assembly Formulation and Cropping

Knowl 4 — Fast Non-Maximum Suppression (Fast NMS)

Knowl 5 — YOLACT Multi-Task Loss and Auxiliary Semantic Head

Knowl 6 — Emergent Translation Variance in Fully Convolutional Prototypes

Knowl 7 — MS COCO Instance Segmentation Benchmark Performance

Knowl 8 — Ablation Analysis of Fast NMS and Prototype Count

Knowl 9 — Mask Boundary Fidelity and Video Temporal Stability

Knowl 10 — YOLACT Failure Modes: Mask Leakage, Localization Failure, and Detector Error

References

Citation

BibTeX

Source Code

Access the Paper