T* And Lv-haystack: A Spatially-guided Temporal Search Framework For Efficient Long-form Video Understanding

Understanding long-form videos—ranging from minutes to hours—presents a awesome situation successful machine vision, particularly arsenic video knowing tasks grow beyond short clips. One of nan cardinal difficulties lies successful efficiently identifying nan fewer applicable frames from thousands wrong a lengthy video basal to reply a fixed query. Most VLMs, specified arsenic LLaVA and Tarsier, process hundreds of tokens per image, making frame-by-frame study of agelong videos computationally expensive. To reside this, a caller paradigm known arsenic temporal hunt has gained prominence. Unlike accepted temporal localization, which typically identifies continuous segments wrong a video, temporal hunt intends to retrieve a sparse group of highly applicable frames dispersed crossed nan full timeline—akin to uncovering a “needle successful a haystack.”

While advancements successful attraction mechanisms and video transformers person improved temporal modeling, these methods still look limitations successful capturing long-range dependencies. Some approaches effort to flooded this by compressing video information aliases selecting circumstantial frames to trim nan input size. Although benchmarks for long-video knowing exist, they mostly measure capacity based connected downstream question-answering tasks alternatively than straight assessing nan effectiveness of temporal search. In contrast, nan emerging attraction connected keyframe action and fine-grained framework retrieval—ranging from glance-based to caption-guided methods—offers a much targeted and businesslike attack to knowing long-form video content.

Stanford, Northwestern, and Carnegie Mellon researchers revisited temporal hunt for long-form video understanding, introducing LV-HAYSTACK—a ample benchmark pinch 480 hours of real-world videos and complete 15,000 annotated QA instances. They framework nan task arsenic uncovering a fewer cardinal frames from thousands, highlighting nan limitations of existent models. To reside this, they propose T, a model that reimagines temporal hunt arsenic a spatial hunt utilizing adaptive zoom-in techniques crossed clip and space. T importantly boosts capacity while reducing computational cost, improving nan accuracy of models for illustration GPT-4o and LLaVA-OV utilizing acold less frames.

The study introduces a Temporal Search (TS) task to heighten video knowing successful long-context ocular connection models. The extremity is to prime a minimal keyframe from a video that retains each accusation basal to reply a fixed question. The projected T model performs this utilizing 3 stages: mobility grounding, iterative temporal search, and task completion. It identifies applicable objects successful nan question, locates them crossed frames utilizing a spatial hunt model, and updates a framework sampling strategy based connected assurance scores. Evaluated connected nan LV-HAYSTACK benchmark, T shows improved ratio and accuracy pinch importantly little computational costs.

The study evaluates nan projected T temporal hunt model crossed aggregate datasets and tasks, including LV-HAYSTACK, LongVideoBench, VideoMME, NExT-QA, EgoSchema, and Ego4D LongVideo QA. T is integrated into open-source and proprietary vision-language models, consistently improving performance, particularly successful agelong videos and constricted framework scenarios. It uses attention, entity detection, aliases trained models for businesslike keyframe selection, achieving precocious accuracy pinch reduced computational cost. Experiments show that T progressively aligns sampling pinch applicable frames complete iterations, approaches human-level capacity pinch much frames, and importantly outperforms azygous and retrieval-based sampling methods crossed various information benchmarks.

In conclusion, nan activity tackles nan situation of knowing long-form videos by revisiting temporal hunt methods utilized successful state-of-the-art VLMs. The authors framework nan task arsenic nan “Long Video Haystack” problem—identifying a fewer applicable frames from tens of thousands. They present LV-HAYSTACK, a benchmark pinch 480 hours of video and complete 15,000 human-annotated instances to support this. Findings show existing methods execute poorly. They propose T, a lightweight model that transforms temporal hunt into a spatial problem utilizing adaptive zooming techniques to reside this. T importantly boosts nan capacity of starring VLMs nether tight framework budgets, demonstrating its effectiveness.

Check out the Paper and Project Page. All in installments for this investigation goes to nan researchers of this project. Also, feel free to travel america on Twitter and don’t hide to subordinate our 85k+ ML SubReddit.

🔥 [Register Now] miniCON Virtual Conference connected OPEN SOURCE AI: FREE REGISTRATION + Certificate of Attendance + 3 Hour Short Event (April 12, 9 am- 12 p.m. PST) + Hands connected Workshop [Sponsored]