arXiv 2610.21459
Can Visual Generation Expand What Unified Multimodal Models Know?
AI13 upvotes
Authors: Wenkai Li, Wenyi Hong, Zhuoyi Yang, Jie Tang
Overview
The authors introduce Knowledge Transfer Probing (KTP), a controlled protocol that teaches models new knowledge through image-generation training and checks whether that knowledge can be accessed through text-only questions. They then add Textual Alignment and Interleaved Image–Text Alignment to connect generation and answering.
Introduction
Unified multimodal models receive image-generation supervision that conveys visual details often missing from text. It is an open question whether learning to generate images can expand what these pretrained models know.
Results
Visual supervision transfers directly to language question answering, but the transfer is uneven across knowledge types. Guidance on selected entities also improves knowledge access on other entities without any QA supervision, with further gains from interleaved alignment.
Conclusion
Visual generation is a source of knowledge beyond text, so training data design and model architecture should be studied together to make that knowledge more broadly accessible.