Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

video-instruction data

Video-instruction data refers to curated multimodal datasets comprising video clips paired with natural language instructions, prompts, or questions alongside their corresponding target responses. This type of data is designed primarily for instruction tuning multimodal large language models and video-language systems, enabling them to comprehend temporal and spatial visual sequences while following open-ended user requests. Typical instances in these datasets encompass tasks such as video summarization, temporal event reasoning, visual question answering, and detailed conversation generation grounded in dynamic visual scenes. By bridging raw video representations with structured linguistic tasks, video-instruction data allows artificial intelligence models to generalize effectively to diverse video-based conversational and analytical interactions.

1 item

Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models

Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models

Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, Fahad Khan

OrganizationsAustralian National UniversityLinköping UniversityMohamed bin Zayed University of Artificial Intelligence

Why you should read this

Presents Video-ChatGPT to enable detailed conversational video understanding by combining a video-adapted visual encoder with a large language model, alongside a 100,000-sample instruction dataset and a standardized quantitative evaluation benchmark.

Conversation agents fueled by Large Language Models (LLMs) are providing a new way to interact with visual data. While there have been initial attempts for image-based conversation models, this work addresses the under-explored field of \emph{video-based conversation} by introducing Video-ChatGPT. It is a multimodal model that merges a video-adapted visual encoder with an LLM. The resulting model is capable of understanding and generating detailed conversations about videos. We introduce a new dataset of 100,000 video-instruction pairs used to train Video-ChatGPT acquired via manual and semi-automated pipeline that is easily scalable and robust to label noise. We also develop a quantitative evaluation framework for video-based dialogue models to objectively analyze the strengths and weaknesses of video-based dialogue models. Code: this https URL.

Added

2026-09-25