MMDU
MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs
Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang, et al. · 2024-10-29
- Modality
- Image
- Data
- Hybrid (real + synthetic/simulated)
- Task · ground truth
- QA/Gen. · Ans.
- Scope
- Broader evaluation with memory demands
What it tests
- CECross-modal evidenceDirectly evaluated
- LELong-range evidencePartially evaluated
- SCState changesNot evaluated
- CFConflicting memoriesNot evaluated
- UQUnanswerable queriesNot evaluated
- MAMemory-grounded actionNot evaluated
- FGFine-grained visual detailPartially evaluated
- ETEntity trackingPartially evaluated
- VCVisual state changeNot evaluated
Survey notes
General multi-image, multi-turn dialogue understanding is scored for six broad response-quality dimensions. The protocol does not separately test retention, revision, or later reuse of experience.