PACO: Parts and Attributes of Common Objects

Vignesh RamanathanAnmol KaliaVladan PetrovicYi WenBaixue ZhengBaishan GuoRui WangAaron MarquezRama KovvuriAbhishek Kadian

article2023CVPR194 citations

Introduces a large-scale visual dataset and benchmark spanning 75 object categories with over 640,000 part masks and detailed attribute annotations across images and videos to advance joint part segmentation and zero-shot instance detection.

Listen

Modern computer vision models increasingly need to provide detailed, fine-grained visual descriptions rather than simply identifying broad object category labels. Applications such as open-vocabulary retrieval and natural language interaction require systems to localize specific components of an object and recognize their detailed characteristics. However, existing public datasets either focus on narrow domains like fashion and birds or lack joint annotations that capture object masks, part masks, and attributes together across a broad set of common objects.

The article introduces PACO (Parts and Attributes of Common Objects), a large-scale benchmark designed to advance research in joint object detection, part segmentation, and attribute recognition across common objects. The objective is to establish an open-source evaluation suite and baseline performance benchmarks across image and video domains for fine-grained object understanding.

To construct PACO, the authors combined image data from the LVIS benchmark with video frames from the Ego4D dataset. The dataset spans 75 common object categories, 456 object-specific part categories, and 55 curated attributes covering colors, patterns, materials, and reflectance properties. In total, the authors collected 641,400 part masks across 260,300 object instances in approximately 81,500 images, making it roughly ten times larger in annotated object instances with parts than previous common object benchmarks. To establish standard reference points, the authors extended standard detection models—specifically Mask R-CNN and Vision Transformer architectures—and evaluated them across three benchmark tasks: part mask segmentation, instance-level attribute prediction, and zero-shot instance detection using multi-attribute queries.

The experimental findings show that segmenting parts and predicting fine-grained attributes are substantially more challenging than standard object detection. On part segmentation, the best Vision Transformer model achieved a mask Average Precision (AP) of 17.7 for object parts compared to 43.4 for whole objects, primarily because object parts are significantly smaller and visually distinct across different parent categories. On instance-level attribute detection, the highest-performing model scored an overall box AP of 19.5 on object attributes and 13.8 on part attributes; performance bounds analysis revealed that the theoretical upper bound with perfect attribute classification exceeds 72 AP, exposing a massive capability gap in current models. In zero-shot instance retrieval, queries with a single attribute yielded higher recall than complex two-attribute queries because prediction errors compounded as query complexity grew. Furthermore, off-the-shelf open-vocabulary detectors struggled significantly on fine-grained queries, registering low single-digit recall scores compared to task-adapted baselines.

These results indicate that current computer vision systems are fundamentally limited in their ability to resolve fine visual details and individual component properties. Improving performance in these areas is crucial for commercial systems that require precise localization, such as inventory management, robotics, and advanced search platforms, where misidentifying part attributes introduces significant operational error. The findings also demonstrate that open-vocabulary foundation models cannot yet reliably substitute for specialized, fine-grained visual detectors without targeted architectural enhancements.

Organizations developing fine-grained perception systems should adopt joint object-part-attribute training objectives and utilize larger vision transformer backbones, which consistently outperformed standard convolutional networks across all benchmark tasks. System architects should also pursue new modeling techniques that reduce compound attribute prediction errors in complex language queries and adapt open-vocabulary frameworks to handle dense attribute reasoning.

While the dataset provides rigorous quality control—including high agreement with expert gold annotations—limitations remain. The vocabulary is constrained to 75 common categories and 55 attributes, and the data distribution exhibits a long-tail imbalance where certain rare parts and attributes have limited examples. Nevertheless, the benchmark provides a highly credible and standardized foundation for evaluating progress in detailed visual understanding.

Meta Simon Fraser University detection algorithm attribute reduction static objects PACO part-level attributes object parts

arXiv: 2301.01795 facebookresearch/paco

Paper: LVIS: A Dataset for Large Vocabulary Instance Segmentation, Agrim Gupta et al. (2019). LVIS provides the core large-vocabulary image benchmark and instance segmentation framework upon which PACO directly builds its part- and attribute-level annotations.
Paper: Ego4D: Around the World in 3,000 Hours of Egocentric Video, Kristen Grauman et al. (2021). Ego4D contributes the foundational egocentric video data that PACO utilizes to annotate object parts and attributes in video domains.
Paper: Describing Objects by their Attributes, Ali Farhadi et al. (2009). This seminal paper introduced the paradigm of recognizing objects via visual attributes and part descriptors that PACO scales to dense modern benchmarks.
Paper: Scene Parsing through ADE20K Dataset, Bolei Zhou et al. (2017). ADE20K pioneered hierarchical scene parsing with multi-level part segmentations, establishing foundational methodologies for evaluating part masks alongside objects.
Paper: Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations, Ranjay Krishna et al. (2016). Visual Genome established the standard for dense crowdsourced object and attribute annotations, motivating PACO's structured part-attribute ontology.
Paper: Mask R-CNN, Kaiming He et al. (2017). Mask R-CNN provides the standard instance segmentation baseline architecture and evaluation metrics extended by PACO for part mask segmentation.
Paper: Masked-attention Mask Transformer for Universal Image Segmentation, Bowen Cheng et al. (2022). Mask2Former represents the modern universal image segmentation architecture evaluated as a primary baseline on PACO's part and instance segmentation benchmarks.
Paper: Open-vocabulary Object Detection via Vision and Language Knowledge Distillation, Xiuye Gu et al. (2021). ViLD establishes vision-language knowledge distillation methods for zero-shot object detection that inform PACO's zero-shot instance detection evaluation.
Paper: Object Detection with Discriminatively Trained Part-Based Models, Pedro F. Felzenszwalb et al. (2010). This paper establishes the foundational computer vision principle of representing objects via deformable configurations of semantic parts.

Paper: Segment Anything, Alexander M. Kirillov et al. (2023). Segment Anything scales promptable mask prediction to a foundation model level, evaluating zero-shot part and object delineation across fine-grained datasets like PACO.

Cover for PACO: Parts and Attributes of Common Objects

Abstract

Object models are gradually progressing from predicting just category labels to providing detailed descriptions of object instances. This motivates the need for large datasets which go beyond traditional object masks and provide richer annotations such as part masks and attributes. Hence, we introduce PACO: Parts and Attributes of Common Objects. It spans 75 object categories, 456 object-part categories and 55 attributes across image (LVIS) and video (Ego4D) datasets. We provide 641K part masks annotated across 260K object boxes, with roughly half of them exhaustively annotated with attributes as well. We design evaluation metrics and provide benchmark results for three tasks on the dataset: part mask segmentation, object and part attribute prediction and zero-shot instance detection. Dataset, models, and code are open-sourced at https://github.com/facebookresearch/paco.

Topics

detection algorithm attribute reduction static objects

Keywords

PACO part-level attributes object parts object attributes attribute recognition few-shot instance detection object detection segmentation benchmark

1. Introduction
1.1. Related work
2. Dataset construction
2.1. Image sources
2.2. Object vocabulary selection
2.3. Parts vocabulary selection
2.4. Attribute vocabulary selection
2.5. Annotation pipeline
2.5.1 Object annotation
2.5.2 Part mask annotation
2.5.3 Attributes annotation
2.5.4 Instance annotation
2.5.5 Managing annotation quality
3. Dataset statistics
4. Tasks and evaluation benchmark
4.1. Dataset splits
4.2. Federated dataset for object categories
4.3. Part segmentation
4.4. Instance-level attributes prediction
4.5. Zero-shot instance detection
5. Benchmarking experiments
5.1. Part segmentation
5.2. Instance-level attributes prediction
5.3. Zero-shot instance detection
6. Conclusion
References

Knowls

Knowl 1 — PACO Dataset Specification and Structure
definition
PACO (Parts and Attributes of Common Objects) is a benchmark dataset designed for the joint tasks of object detection, part mask segmentation, and instance-level attribute recognition across both image and video domains.

The dataset is characterized by the following specifications:
- Object Categories: 75 common object categories selected from the intersection of frequent LVIS categories and Ego4D video narrations having at least 20 instances in Ego4D.
- Object-Part Categories: 456 object-specific part categories derived from 200 object-agnostic part classes mined from web diagrams and curated to ensure visibility and distinguishability across instances.
- Attributes: 55 visual attributes grouped into four categories: 29 colors, 10 patterns and markings, 13 materials, and 3 reflectance levels.
- Image Sources and Annotation Counts:
  - PACO-LVIS (static images): 57.6K images with object masks, 52.7K images with part masks, 274K object masks, 502K part masks, 74.4K objects with attributes, and 186K parts with attributes.
  - PACO-EGO4D (egocentric video frames): 23.9K frames with object masks, 24K frames with part masks, 58.4K object masks, 139.3K part masks, 49.6K objects with attributes, and 110.6K parts with attributes.
  - Joint Totals: 641.4K part masks across 260.3K object instances in 76.7K images/frames. Attribute annotations cover 124K object instances and 296.6K part instances, exhaustively annotated across all 55 attributes per annotated box (averaging 23.4 attribute annotations per image) and providing explicit negative attribute labels.
- Dataset Splits:
  - PACO-LVIS: 45,790 train, 2,410 validation, and 9,443 test images (the test split is a strict subset of the LVIS-v1 validation set).
  - PACO-EGO4D: 15,667 train, 825 validation, and 9,892 test frames, with mutually disjoint object instance IDs between train and test sets (16,908 unique object instances annotated in total).
Knowl 2 — Federated Evaluation Protocol for Object-Part and Attribute Detection
model/method
Because PACO is a federated dataset where not every image is exhaustively annotated for all categories and attributes, evaluation metrics adapt the Average Precision (AP) protocol across Intersection over Union (IoU) thresholds to handle missing labels.

1. Part Segmentation ( $AP^{opart}$ )

For an object-part category $(o, p)$ combining object $o$ and part $p$ :
- Negative images of object $o$ : All predicted masks for $(o, p)$ are marked as false positives.
- Non-exhaustive positive images: Predicted masks overlapping an annotated ground-truth $(o, p)$ mask above the IoU threshold are true positives. All other predicted masks are ignored.
- Exhaustive positive images: Predicted masks matching ground-truth $(o, p)$ masks above the IoU threshold are true positives; unmatched predictions are false positives.
2. Instance-Level Category-Attribute Detection ( $AP^{obj}_{att}$ and $AP^{opart}_{att}$ )

For a category $c \in \{o, (o, p)\}$ and an attribute $a$ , AP is computed only if the evaluation split contains $\ge 1$ positive instance of $(c, a)$ and $\ge 40$ negative instances of category $c$ for attribute $a$ :
- Negative images of object $o$ : All predicted masks for $(c, a)$ are false positives.
- Positive images (both exhaustive and non-exhaustive):
  - Predictions overlapping ground-truth category $c$ annotated positively for $a$ are true positives.
  - Predictions overlapping ground-truth category $c$ annotated negatively for $a$ are false positives.
  - Predictions overlapping ground-truth category $c$ with un-annotated attribute labels are ignored.
  - Predictions not overlapping any ground-truth mask of category $c$ are ignored in non-exhaustive images and marked as false positives in exhaustive images.
To reduce metric variance from rare category-attribute combinations, AP values for $(c, a)$ are averaged across all valid categories $c$ for attribute $a$ to yield $AP_a$ . These are subsequently averaged across attribute types: color ( $AP_{col}$ ), pattern & markings ( $AP_{pat}$ ), material ( $AP_{mat}$ ), reflectance ( $AP_{ref}$ ), and overall ( $AP_{att}$ ).
Knowl 3 — Zero-Shot Instance Detection Task and Query Hierarchy
definition
Zero-shot instance detection is the task of retrieving and localizing the bounding box of a specific instance of an object from an image collection using a compositional language query, where no sample images of that instance have been previously seen.

Query Hierarchy

Queries are structured by compositionality into level- $k$ ( $Lk$ ) queries containing $k$ attributes of the object and/or its parts:
- Level 1 ( $L1$ ): 1 attribute (e.g., "blue mug" or "mug with a blue handle").
- Level 2 ( $L2$ ): 2 attributes (e.g., "blue striped mug").
- Level 3 ( $L3$ ): 3 attributes (e.g., "blue striped mug with white handle").
Evaluation Setup and Metrics
- Target and Distractors: Each query is evaluated against 1 positive target image and up to 100 distractor images. Distractor images contain hard negatives: other instances of the same object class that differ by at least one attribute from the query. On average, $>40\%$ of distractors per query are hard negatives.
- Query Statistics: PACO-LVIS contains 931 $L1$ , 2,348 $L2$ , and 2,000 $L3$ queries. PACO-EGO4D contains 793 $L1$ , 1,437 $L2$ , and 2,115 $L3$ queries.
- Metric: Average Recall at top- $k$ ( $AR@k$ for $k \in \{1, 5\}$ ), calculated across bounding box IoU thresholds from $0.5$ to $0.95$ .
Knowl 4 — Joint Object, Part, and Attribute Detector Architecture and Scoring
model/method
Joint detection of objects, parts, and attributes extends standard two-stage object detectors (Mask R-CNN and ViT-det) with multi-task prediction heads:
1. Object and Part Detection Head: The standard classification head is configured to predict 531 mutually distinct classes (75 object categories plus 456 object-part categories) along with bounding boxes and segmentation masks. Models are trained using federated cross-entropy loss and Large Scale Jittering (LSJ) data augmentation.
2. Attribute Head: An auxiliary multi-task head attached to the shared detector backbone uses the same Region of Interest (RoI)-pooled features as the detection head to predict object and part attributes. It outputs independent multi-class probability distributions for each of the four attribute types (color, pattern, material, reflectance) optimized via separate cross-entropy loss terms.
3. Inference Ranking:
  - For a single category-attribute pair $(c, a)$ where $c$ is an object or object-part and $a$ is an attribute, candidate bounding boxes are ranked by the product of their category score $S_c$ and attribute score $S_a$ : $S(c, a) = S_c \times S_a$
  - For zero-shot instance queries composed of multiple object, part, and attribute terms, candidate bounding boxes are ranked using the geometric mean of the constituent detection and attribute probabilities predicted by the model.

Knowl 5 — Object and Object-Part Segmentation Benchmark Results on PACO-LVIS

data/table

Instance segmentation performance for object categories ( $AP^{obj}$ ) and object-part categories ( $AP^{opart}$ ) on the PACO-LVIS test split across ResNet and Vision Transformer (ViT-det) backbones with Feature Pyramid Networks (FPN) and Cascade FPN:

Model	Mask $AP^{obj}$	Mask $AP^{opart}$	Box $AP^{obj}$	Box $AP^{opart}$
ResNet-50 FPN	$31.5 \pm 0.3$	$12.3 \pm 0.1$	$34.6 \pm 0.3$	$16.0 \pm 0.1$
ResNet-50 Cascade FPN	$32.6 \pm 1.3$	$12.5 \pm 0.7$	$37.4 \pm 1.6$	$16.3 \pm 1.1$
ResNet-101 FPN	$31.5 \pm 0.6$	$12.3 \pm 0.3$	$34.8 \pm 0.8$	$16.1 \pm 0.3$
ResNet-101 Cascade FPN	$35.1 \pm 0.1$	$13.7 \pm 0.1$	$40.2 \pm 0.1$	$17.9 \pm 0.2$
ViT-B FPN	$33.6 \pm 0.3$	$13.5 \pm 0.1$	$38.7 \pm 0.4$	$17.5 \pm 0.0$
ViT-B Cascade FPN	$33.6 \pm 0.3$	$13.5 \pm 0.1$	$38.7 \pm 0.4$	$17.5 \pm 0.0$
ViT-L FPN	$42.8 \pm 0.3$	$17.3 \pm 0.1$	$47.3 \pm 0.2$	$22.0 \pm 0.1$
ViT-L Cascade FPN	$43.4 \pm 0.3$	$17.7 \pm 0.0$	$49.7 \pm 0.2$	$22.9 \pm 0.0$

Object-part segmentation ( $AP^{opart}$ ) is significantly lower than object segmentation ( $AP^{obj}$ ) across all models (e.g., $17.7\%$ vs. $43.4\%$ mask AP for ViT-L Cascade FPN). This degradation is primarily driven by the smaller physical size distribution of parts relative to full objects. Scaling the backbone architecture to ViT-L provides consistent improvements across both tasks.

Knowl 6 — Instance-Level Attribute Prediction Results and Oracle Bounds on PACO-LVIS

data/table

Box Average Precision (Box AP) performance for object attributes ( $AP^{obj}$ ) and object-part attributes ( $AP^{opart}$ ) on the PACO-LVIS test split, categorized by overall attribute score ( $att$ ), color ( $col$ ), pattern/marking ( $pat$ ), material ( $mat$ ), and reflectance ( $ref$ ):

Model	$AP^{obj}_{att}$	$AP^{obj}_{col}$	$AP^{obj}_{pat}$	$AP^{obj}_{mat}$	$AP^{obj}_{ref}$	$AP^{opart}_{att}$	$AP^{opart}_{col}$	$AP^{opart}_{pat}$	$AP^{opart}_{mat}$	$AP^{opart}_{ref}$
R50 FPN	$13.5 \pm 0.3$	$10.8 \pm 0.1$	$14.1 \pm 0.6$	$9.9 \pm 0.4$	$19.1 \pm 0.7$	$9.7 \pm 0.2$	$10.7 \pm 0.2$	$10.6 \pm 0.5$	$6.9 \pm 0.0$	$10.7 \pm 0.2$
R50 Cascade	$15.0 \pm 1.0$	$12.4 \pm 0.7$	$16.1 \pm 0.7$	$11.0 \pm 0.9$	$20.6 \pm 1.6$	$10.5 \pm 0.7$	$11.6 \pm 0.8$	$11.6 \pm 0.8$	$7.6 \pm 0.7$	$11.2 \pm 0.7$
R101 FPN	$13.5 \pm 0.3$	$11.0 \pm 0.2$	$13.9 \pm 0.3$	$9.9 \pm 0.4$	$19.1 \pm 0.6$	$9.9 \pm 0.1$	$11.0 \pm 0.4$	$10.8 \pm 0.4$	$7.1 \pm 0.2$	$10.9 \pm 0.3$
R101 Cascade	$16.0 \pm 0.1$	$13.4 \pm 0.2$	$16.7 \pm 0.2$	$12.3 \pm 0.1$	$21.5 \pm 0.4$	$11.5 \pm 0.2$	$12.6 \pm 0.1$	$12.5 \pm 0.3$	$8.5 \pm 0.3$	$12.6 \pm 0.3$
ViT-B FPN	$15.0 \pm 0.2$	$11.9 \pm 0.1$	$14.9 \pm 0.5$	$12.8 \pm 0.4$	$20.4 \pm 0.8$	$10.9 \pm 0.2$	$11.3 \pm 0.3$	$11.4 \pm 0.6$	$9.0 \pm 0.1$	$11.8 \pm 0.3$
ViT-B Cascade	$15.7 \pm 0.2$	$12.6 \pm 0.1$	$16.0 \pm 0.5$	$13.2 \pm 0.4$	$20.9 \pm 0.5$	$11.0 \pm 0.2$	$11.6 \pm 0.2$	$11.7 \pm 0.4$	$9.0 \pm 0.2$	$11.5 \pm 0.3$
ViT-L FPN	$18.8 \pm 0.3$	$14.9 \pm 0.2$	$18.9 \pm 1.0$	$16.0 \pm 0.7$	$25.4 \pm 0.7$	$13.5 \pm 0.2$	$14.0 \pm 0.2$	$14.0 \pm 0.4$	$11.7 \pm 0.4$	$14.3 \pm 0.6$
ViT-L Cascade	$19.5 \pm 0.3$	$15.6 \pm 0.3$	$19.1 \pm 0.5$	$16.3 \pm 0.3$	$27.0 \pm 0.4$	$13.8 \pm 0.1$	$14.4 \pm 0.3$	$15.1 \pm 0.0$	$11.5 \pm 0.2$	$14.5 \pm 0.4$

Oracle Sensitivity Bounds on Attribute Prediction

To isolate attribute classification accuracy from object detection quality, $AP^{obj}_{att}$ bounds are computed by freezing the model's box detections and replacing predicted attribute scores with (1) Lower Bound (LB): uniform attribute scores (ignoring attribute predictions), and (2) Upper Bound (UB): ground-truth attribute oracle scores:

Model	LB (No Attribute)	Original	UB (Perfect Attribute)
ResNet-50 FPN	$8.6 \pm 0.3$	$13.5 \pm 0.3$	$61.4 \pm 0.3$
ResNet-101 FPN	$8.6 \pm 0.3$	$13.5 \pm 0.3$	$63.0 \pm 0.3$
ViT-B FPN	$9.0 \pm 0.1$	$15.0 \pm 0.2$	$60.5 \pm 0.1$
ViT-L FPN	$10.6 \pm 0.2$	$18.8 \pm 0.3$	$72.6 \pm 0.3$

The substantial gap between baseline model performance ( $13.5\% - 18.8\%$ ) and the upper bound ( $60.5\% - 72.6\%$ ) demonstrates that attribute classification errors constitute the primary bottleneck in joint instance-attribute prediction.

Knowl 7 — Zero-Shot Instance Detection Performance Across Query Complexity Levels

data/table

Evaluation of FPN models on the PACO-LVIS zero-shot instance detection test set across query complexity levels ( $L1$ , $L2$ , $L3$ , and all queries combined) using Average Recall at top-1 ( $AR@1$ ) and top-5 ( $AR@5$ ):

	L1 queries		L2 queries		L3 queries		All queries
Model	$AR@1$	$AR@5$	$AR@1$	$AR@5$	$AR@1$	$AR@5$	$AR@1$	$AR@5$
ResNet-50 FPN	$22.5 \pm 0.7$	$39.2 \pm 0.5$	$20.1 \pm 0.4$	$38.5 \pm 0.1$	$22.3 \pm 0.9$	$44.5 \pm 1.1$	$21.4 \pm 0.6$	$40.9 \pm 0.3$
ResNet-101 FPN	$23.1 \pm 0.7$	$40.5 \pm 1.4$	$20.0 \pm 0.6$	$39.3 \pm 1.0$	$23.1 \pm 0.7$	$45.2 \pm 0.6$	$21.7 \pm 0.6$	$41.8 \pm 0.8$
ViT-B FPN	$26.8 \pm 0.2$	$45.8 \pm 0.2$	$22.7 \pm 0.5$	$40.0 \pm 0.7$	$24.1 \pm 0.5$	$42.5 \pm 1.5$	$23.9 \pm 0.4$	$42.0 \pm 0.9$
ViT-L FPN	$35.3 \pm 0.7$	$57.3 \pm 0.6$	$29.7 \pm 0.6$	$50.1 \pm 0.2$	$31.1 \pm 0.8$	$52.3 \pm 0.9$	$31.2 \pm 0.4$	$52.2 \pm 0.5$

Across all model architectures, performance exhibits a consistent non-monotonic trend: $L1 > L3 > L2$ . This trend reflects the trade-off between two countervailing effects:

Information gain: Highly compositional queries ( $L3$ ) provide more descriptive constraints, making target instance retrieval easier than moderately compositional queries ( $L2$ ).
Error compounding: Evaluating multiple attribute predictions simultaneously increases the likelihood of classification failure, causing single-attribute queries ( $L1$ ) to outperform multi-attribute queries ( $L3$ ).

Knowl 8 — Zero-Shot Instance Detection Comparison: Open-Vocabulary Detectors vs PACO Baselines

data/table

Zero-shot instance detection evaluation on PACO-LVIS comparing off-the-shelf open-vocabulary detectors (MDETR and Detic) against models trained directly on PACO for Level 1 queries ( $L1$ ), broken down by queries containing only object attributes ( $L1_{obj}$ ) and only part attributes ( $L1_{part}$ ):

Model	$L1_{obj}$	$L1_{part}$	All $L1$
MDETR (ResNet-101)	$4.1 \pm 0.6$	$5.3 \pm 0.6$	$4.9 \pm 0.3$
PACO ResNet-101 FPN	$20.3 \pm 0.9$	$24.4 \pm 1.0$	$23.1 \pm 0.7$
Detic (Swin-B)	$5.2 \pm 0.7$	$6.2 \pm 0.3$	$5.9 \pm 0.2$
PACO ViT-B FPN	$22.6 \pm 0.8$	$28.9 \pm 0.6$	$26.8 \pm 0.2$

All numbers report $AR@1$ . Open-vocabulary detectors achieve low recall ( $4.9\%$ for MDETR and $5.9\%$ for Detic on All $L1$ ) compared to PACO-trained baselines ( $23.1\%$ and $26.8\%$ ) due to architectural and training limitations:

Detic is primarily trained on noun vocabularies and has limited fine-grained attribute grounding capability.
MDETR is trained for referring expressions on positive images and exhibits poor discrimination against hard negative distractor images.

Knowl 9 — Zero-Shot vs Few-Shot Instance Detection Discrepancy on PACO-EGO4D
empirical result
On the PACO-EGO4D benchmark, where multiple video frames depict the same physical object instance, few-shot instance detection was compared against zero-shot attribute-based instance detection across 1,992 queries that have at least 6 bounding box instances of the target object.
- Few-Shot Model Setup: A two-stage baseline employing a pre-trained ResNet-50 FPN object detector followed by nearest-neighbor ranking of RoI-pooled features against $k$ visual exemplar bounding boxes ( $k \in \{1, 2, 3, 4, 5\}$ ).
- Performance Gap:
  - Zero-shot FPN detectors trained on the joint PACO dataset achieve between $15\%$ and $22\%$ $AR@1$ (with ResNet-101 FPN achieving $\approx 22\%$ , ViT-B FPN $\approx 17\%$ , and ResNet-50 FPN $\approx 15\%$ ).
  - The 1-shot visual baseline ( $k = 1$ ) achieves over $42\%$ $AR@1$ , creating an absolute margin of $>20$ points over the strongest zero-shot model.
  - Scaling visual exemplars from $k = 1$ to $k = 5$ further elevates few-shot retrieval performance toward $50\%$ .
This indicates that current attribute and part language descriptions in zero-shot models are far less effective at capturing instance-level identity than even a single visual bounding box exemplar.

Coverage note — No substantial contributed material was omitted; all key definitions, dataset statistics, federated metric formulations, baseline architectures, and experimental results from the main paper have been fully captured.

References

1.Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6154–6162, 2018. 7
2.Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1971–1978, 2014. 1, 2
3.Zhenfang Chen, Peng Wang, Lin Ma, Kwan-Yee K. Wong, and Qi Wu. Cops-ref: A new dataset and task on compositional referring expression comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. 1, 3, 6
4.Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 2
5.Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6):141–142, 2012. 2
6.N. Dinesh Reddy, Minh Vo, and Srinivasa G. Narasimhan. Carfusion: Combining point tracking and part detection for dynamic 3d reconstruction of vehicles. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018. 2
7.M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1):98–136, Jan. 2015. 2
8.Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Promptdet: Expand your detector vocabulary with uncurated images. arXiv preprint arXiv:2203.16513, 2022. 1
9.Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2918–2928, 2021. 7
10.Ke Gong, Xiaodan Liang, Dongyu Zhang, Xiaohui Shen, and Liang Lin. Look into person: Self-supervised structuresensitive learning and a new benchmark for human parsing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 932–940, 2017. 2
11.Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995–19012, 2022. 2, 3
12.Sheng Guo, Weilin Huang, Xiao Zhang, Prasanna Srikhanta, Yin Cui, Yuan Li, Hartwig Adam, Matthew R Scott, and Serge Belongie. The imaterialist fashion attribute dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019. 2
13.Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019. 2, 3, 5, 6
14.Akshita Gupta, Sanath Narayan, KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Ow-detr: Open-world detection transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9235–9244, 2022. 1
15.Ju He, Shuo Yang, Shaokang Yang, Adam Kortylewski, Xiaoding Yuan, Jie-Neng Chen, Shuai Liu, Cheng Yang, and Alan Yuille. Partimagenet: A large, high-quality dataset of parts. arXiv preprint arXiv:2112.00933, 2021. 1, 2
16.Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017. 2
17.Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019. 1, 3
18.Menglin Jia, Mengyun Shi, Mikhail Sirotenko, Yin Cui, Claire Cardie, Bharath Hariharan, Hartwig Adam, and Serge Belongie. Fashionpedia: Ontology, segmentation, and an attribute localization dataset. In European conference on computer vision, pages 316–332. Springer, 2020. 1, 2, 3
19.Licheng Jiao, Ruohan Zhang, Fang Liu, Shuyuan Yang, Biao Hou, Lingling Li, and Xu Tang. New generation deep learning for video object detection: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2021. 2
20.Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetrmodulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780–1790, 2021. 1, 8
21.Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014. 1, 3, 6
22.Ronak Kosti, Jose M Alvarez, Adria Recasens, and Agata Lapedriza. Emotion recognition in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1667–1675, 2017. 2
23.Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32–73, 2017. 1
24.Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision, 128(7):1956–1981, 2020. 2
25.Dangwei Li, Zhang Zhang, Xiaotang Chen, Haibin Ling, and Kaiqi Huang. A richly annotated dataset for pedestrian attribute recognition. arXiv preprint arXiv:1603.07054, 2016. 1
26.Jianshu Li, Jian Zhao, Yunchao Wei, Congyan Lang, Yidong Li, Terence Sim, Shuicheng Yan, and Jiashi Feng. Multiplehuman parsing in the wild. arXiv preprint arXiv:1705.07206, 2017. 2
27.Yining Li, Chen Huang, Chen Change Loy, and Xiaoou Tang. Human attribute recognition by deep hierarchical contexts. In European conference on computer vision, pages 684–700. Springer, 2016. 2
28.Yanghao Li, Hanzi Mao, Ross B. Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. ArXiv, abs/2203.16527, 2022. 2, 7
29.Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017. 7
30.Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. 2
31.Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1096–1104, 2016. 2
32.Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016. 1, 3, 6
33.Panagiotis Meletis, Xiaoxiao Wen, Chenyang Lu, Daan de Geus, and Gijs Dubbelman. Cityscapes-panoptic-parts and pascal-panoptic-parts datasets for scene understanding. arXiv preprint arXiv:2004.07944, 2020. 2
34.Genevieve Patterson and James Hays. Coco attributes: Attributes for people, animals, and objects. European Conference on Computer Vision, 2016. 1
35.Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivastava. Learning to predict visual attributes in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13018–13028, 2021. 1, 2
36.Bernardino Romera-Paredes and Philip Torr. An embarrassingly simple approach to zero-shot learning. In International conference on machine learning, pages 2152–2161. PMLR, 2015. 3
37.Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8430–8439, 2019. 2
38.Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15638–15650, 2022. 7
39.Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojciech Galuba, and Tal Hassner. Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8802–8812, 2021. 2
40.Xibin Song, Peng Wang, Dingfu Zhou, Rui Zhu, Chenye Guan, Yuchao Dai, Hao Su, Hongdong Li, and Ruigang Yang. Apollocar3d: A large 3d car instance understanding benchmark for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5452–5462, 2019. 2
41.Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.07140, 2016. 2, 4, 5, 6, 7
42.Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. Technical report, 2011. 1, 2
43.Chenyun Wu, Zhe Lin, Scott Cohen, Trung Bui, and Subhransu Maji. Phrasecut: Language-based image segmentation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10216–10225, 2020. 3
44.Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning-the good, the bad and the ugly. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4582–4591, 2017. 3
45.Ke Yan, Youbao Tang, Yifan Peng, Veit Sandfort, Mohammadhadi Bagheri, Zhiyong Lu, and Ronald M Summers. Mulan: multitask universal lesion analysis network for joint lesion detection, tagging, and segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 194–202. Springer, 2019. 2
46.Lu Yang, Qing Song, Zhihui Wang, and Ming Jiang. Parsing r-cnn for instance-level human analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 364–373, 2019. 2
47.Shuai Zheng, Fan Yang, M Hadi Kiapour, and Robinson Piramuthu. Modanet: A large-scale street fashion dataset with polygon annotations. In Proceedings of the 26th ACM international conference on Multimedia, pages 1670–1678, 2018. 1, 2
48.Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452–1464, 2017. 2
49.Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633–641, 2017. 2
50.Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):302–321, 2019. 1, 2
51.Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krahenb ü uhl, and Ishan Misra. Detecting twenty-thousand ¨ classes using image-level supervision. arXiv preprint arXiv:2201.02605, 2022. 1, 8
52.Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020. 7

Citation

MLA

Ramanathan, V., et al. “PACO: Parts and Attributes of Common Objects”. arXiv, 2023, http://arxiv.org/abs/2301.01795v1.

APA

Ramanathan, V., Kalia, A., Petrovic, V., Wen, Y., Zheng, B., Guo, B., Wang, R., Marquez, A., Kovvuri, R., Kadian, A., Mousavi, A., Song, Y., Dubey, A., & Mahajan, D. (2023). PACO: Parts and Attributes of Common Objects. arXiv. http://arxiv.org/abs/2301.01795v1

Chicago

Ramanathan, V., A. Kalia, V. Petrovic, et al. 2023. “PACO: Parts and Attributes of Common Objects”. arXiv. http://arxiv.org/abs/2301.01795v1.

Harvard

Ramanathan, V. et al. (2023) “PACO: Parts and Attributes of Common Objects”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2301.01795v1.

Vancouver

1. Ramanathan V, Kalia A, Petrovic V, et al (2023) PACO: Parts and Attributes of Common Objects. arXiv

BibTeX

@article{ramanathan2023paco,
  title = {PACO: Parts and Attributes of Common Objects},
  author = {Ramanathan, Vignesh and Kalia, Anmol and Petrovic, Vladan and Wen, Yi and Zheng, Baixue and Guo, Baishan and Wang, Rui and Marquez, Aaron and Kovvuri, Rama and Kadian, Abhishek and Mousavi, Amir and Song, Yiwen and Dubey, Abhimanyu and Mahajan, Dhruv},
  year = {2023},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2301.01795v1},
  eprint = {2301.01795}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/publicdomain/zero/1.0/

PACO: Parts and Attributes of Common Objects

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — PACO Dataset Specification and Structure

Knowl 2 — Federated Evaluation Protocol for Object-Part and Attribute Detection

1. Part Segmentation (APopartAP^{opart}APopart)

2. Instance-Level Category-Attribute Detection (APattobjAP^{obj}_{att}APattobj​ and APattopartAP^{opart}_{att}APattopart​)

Knowl 3 — Zero-Shot Instance Detection Task and Query Hierarchy

Query Hierarchy

Evaluation Setup and Metrics

Knowl 4 — Joint Object, Part, and Attribute Detector Architecture and Scoring

Knowl 5 — Object and Object-Part Segmentation Benchmark Results on PACO-LVIS

Knowl 6 — Instance-Level Attribute Prediction Results and Oracle Bounds on PACO-LVIS

Oracle Sensitivity Bounds on Attribute Prediction

Knowl 7 — Zero-Shot Instance Detection Performance Across Query Complexity Levels

Knowl 8 — Zero-Shot Instance Detection Comparison: Open-Vocabulary Detectors vs PACO Baselines

Knowl 9 — Zero-Shot vs Few-Shot Instance Detection Discrepancy on PACO-EGO4D

References

Citation

BibTeX

Source Code

Access the Paper

1. Part Segmentation ( $AP^{opart}$ )

2. Instance-Level Category-Attribute Detection ( $AP^{obj}_{att}$ and $AP^{opart}_{att}$ )