What happened
Multimodal large language models often rely on visual chain-of-thought (Visual CoT), which generates intermediate reasoning images to handle spatial, temporal, and embodied tasks.
While Visual CoT aids visual foresight, it adds significant inference overhead, especially for proactive video reasoning.
Apple's new framework, Internalized Visual Thinking (IVT), shifts visual reasoning to training time, allowing direct reasoning at inference without intermediate image generation.
Why it matters
This approach could make video reasoning more efficient and practical for real-time applications, reducing computational burden.
By internalizing visual thinking, models may achieve faster inference without sacrificing reasoning quality, potentially enabling broader deployment.
Key facts
Visual CoT generates intermediate reasoning images for spatial, temporal, and embodied reasoning.
Visual CoT introduces substantial inference overhead.
IVT is a post-training framework that jointly optimizes textual prediction and visual thinking.
IVT aims to learn visual thinking during training and reason directly at inference.
What to watch next
Whether IVT can maintain or improve reasoning accuracy compared to Visual CoT in real-world video tasks.
How IVT's training efficiency and scalability compare to existing methods in large-scale deployments.