EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations

Existing computer vision models largely rely on static images or short video clips, struggling to capture how objects physically transform over extended interactions. In real-world environments like kitchens, objects undergo dynamic state changes—such as whole vegetables being peeled, sliced, and cooked into composite meals—while being actively manipulated by human hands. Past video datasets provided either short-term object segmentations without detailed action labels or broad action labels limited to coarse bounding boxes. The article addresses this gap by introducing a benchmark suite and dataset designed to capture precise pixel-level object transformations, hand-object contacts, and long-term entity tracking over time.

The main objective of the article is to establish a comprehensive pixel-level annotation pipeline and benchmark suite for first-person (egocentric) video. Specifically, the article evaluates object tracking, hand-object contact segmentation, and long-term source retrieval using untrimmed videos from the EPIC-KITCHENS collection.

To accomplish this, the authors created an interactive annotation pipeline that combined artificial intelligence segmentation tools with manual human verification over a 22-month period. Working across 36 hours of video from 179 recordings, the team generated 271,600 manual masks across 257 object classes, which were further expanded to 9.9 million dense masks through automated bidirectional interpolation. The dataset was structured into sequences averaging 12 seconds across consecutive actions, which is 2.5 to 4 times longer than previous benchmark datasets. The authors established three distinct evaluation challenges: tracking object masks across consecutive actions, segmenting hands alongside the specific objects they contact, and tracing a query object back across minutes of video to identify its source container.

The experimental findings demonstrate significant performance variations across tasks. Fine-tuning models directly on this dataset improved video object segmentation accuracy by roughly 13% over general pre-trained models, achieving a tracking score of 78.0 on the test set. However, performance dropped by about 7 to 9 points when evaluating in previously unseen kitchen environments. Hand detection and segmentation proved highly accurate, reaching over 90% accuracy, but segmenting manipulated objects and predicting contact states remained challenging, scoring between 24% and 34%. For long-term reasoning, baseline models struggled to locate when an object emerged from its source container across average time gaps of 5.4 minutes, but providing oracle temporal boundaries boosted source identification accuracy from 34% to over 94%.

These results indicate that while current computer vision architectures reliably detect hands and track rigid objects, they face severe limitations when tracking severe physical transformations, occlusions, and long-term scene interactions. Developing robust egocentric vision systems is critical for applications in robotics, assistive augmented reality, and workplace monitoring, where systems must maintain spatial awareness of tools and changing materials. The substantial drop in performance within unseen environments highlights the ongoing operational risk of deploying vision systems that fail to generalize across diverse physical settings.

Moving forward, researchers and developers should focus on multi-frame reasoning architectures that can infer contact states and locate temporal transitions rather than relying on single-frame detection. Teams should leverage the publicly available dataset and code repositories to evaluate how foundation models handle complex physical transformations over extended horizons.

The findings are constrained by several dataset limitations, including severe motion blur in first-person cameras, long-tailed distributions where common containers like refrigerators dominate source reasoning, and annotator ambiguities regarding complex boundaries like sauces and mixtures. Furthermore, benchmark evaluations for long-term reasoning relied on oracle-assisted baselines rather than fully automated end-to-end artificial intelligence systems, meaning practical deployments will face greater real-world performance degradation until temporal reasoning capabilities improve.