Elamur Research⌕
☰
home / papers / 2026 / Not All Modalities at Once: Dynamic Dropout and Bidirectional Fusion for Robust Multi-modal Knowledge Graph Completion

Not All Modalities at Once: Dynamic Dropout and Bidirectional Fusion for Robust Multi-modal Knowledge Graph Completion

Paper · 2026 · Jiashun Peng, Fu Zhang, Hongzhi Chen et al.

Summary

Multi-modal knowledge graph completion (MKGC) predicts missing entities in triples by combining graph structure with entity descriptions and images. The authors argue that training on every modality at once lets models lean too heavily on a single auxiliary source and makes unevenly noisy modalities compete. Their framework, MDBGF, always keeps the structural embedding. In each epoch it may randomly hide either the text or the image input, with a hiding probability that rises linearly during training up to a cap α. In epochs where all inputs are kept, a bidirectional gated fusion (BGF) module lets textual and visual features rescale each other's fusion weights through learnable gates. Training also uses AHNS, a negative sampler that mixes in-batch random negatives with high-scoring hard negatives in an adaptively tuned ratio. Across three benchmarks, MDBGF is reported to outperform prior systems and set new state-of-the-art results. The dropout schedule also improves several other backbone models. This matters for real knowledge graphs, where entity images or texts are often missing or noisy.

Key points

  • Three components: dynamic modality dropout (MD), BGF for text–vision interaction, and AHNS for hard-negative mining. Prediction uses a Tucker scoring function, and visual and textual tokens are filtered by frequency, following MyGO.
  • Missing-modality robustness on MKG-W: the authors mask modalities at test time, since no official missing-modality benchmark exists. Performance falls only about 1–2 points:
    • MRR drops from 40.10 to 38.19.
    • Hits@1 drops from 33.42 to 31.76.
    • Hits@10 drops from 52.35 to 50.35.
  • Modality interference: with MD and BGF removed, the baseline on MKG-W scores higher with structure+text (MRR 38.9) than with all three modalities (37.3). This suggests redundant visual features can hurt, which motivates the dropout mechanism.
  • Memory efficiency: AHNS needs 21286M of memory, compared with 39551M for MyGO's contrastive sampler (FGCL) and 53350M for AdaMF's noise-based sampler. AHNS also gives better accuracy than both.
  • Resources: code is released at github.com/ferryman-ship/MDBGF. MD adds negligible memory, while BGF is the main source of overhead.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Northeastern UniversityShenyang · China42 works · 1,940 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 657 worksKnowledge graphs & ontologies · 192 works

Cite

Cite as Jiashun Peng, Fu Zhang, Hongzhi Chen, Jingwei Cheng, Yingsong Ning and Xiaoke Wang. 2026. Not All Modalities at Once: Dynamic Dropout and Bidirectional Fusion for Robust Multi-modal Knowledge Graph Completion. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.890

@inproceedings{peng2026not,
  title = {{Not All Modalities at Once: Dynamic Dropout and Bidirectional Fusion for Robust Multi-modal Knowledge Graph Completion}},
  author = {Jiashun Peng, Fu Zhang, Hongzhi Chen, Jingwei Cheng, Yingsong Ning and Xiaoke Wang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.890},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.890}
}

References

26 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ivana Balažević, Carl Allen, and Timothy Hospedales. 2019. Tucker: Tensor factorization for knowledge graph completion. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP),…
  2. Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems (NeurIPS), 26.
  3. Jaejun Lee, Chanyoung Chung, Hochang Lee, Sungho Jo, and Joyce Whang. 2023. Vista: Visual-textual knowledge graph representation learning. In Findings of the association for computational linguistics: EMNLP 2023, pages 7314–7328. find
  4. Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, and 1 others. 2015. Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167–195.
  5. Xinhang Li, Xiangyu Zhao, Jiaxing Xu, Yong Zhang, and Chunxiao Xing. 2023. Imf: interactive multimodal fusion model for link prediction. In Proceedings of the ACM web conference (WWW), pages 2572–2580. find
  6. Ye Liu, Hui Li, Alberto Garcia-Duran, Mathias Niepert, Daniel Onoro-Rubio, and David S Rosenblum. 2019. Mmkg: multi-modal knowledge graphs. In European Semantic Web Conference (ESWC), pages 459–474. Springer. find
  7. Zongsheng Cao, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. 2022. Otkge: Multi-modal knowledge graph embeddings via optimal transport. Advances in neural information processing systems (NeurIPS), 35:39090–39102. find
  8. Hatem Mousselly-Sergieh, Teresa Botschen, Iryna Gurevych, and Stefan Roth. 2018. A multimodal translation-based approach for knowledge graph representation learning. In Proceedings of the seventh joint conference on lexical and computational semantics (SEM), pages 225–234. find
  9. Zhuo Chen, Yichi Zhang, Yin Fang, Yuxia Geng, Lingbing Guo, Xiang Chen, Qian Li, Wen Zhang, Jiaoyan Chen, Yushan Zhu, and 1 others. 2024. Knowledge graphs meet multi-modal learning: A comprehensive survey. arXiv preprint arXiv:2402.05391. — [arXiv:2402.05391](https://arxiv.org/abs/2402.05391) find
  10. Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. 2022. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208.06366. — [arXiv:2208.06366](https://arxiv.org/abs/2208.06366) find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,…
  12. Yuxiao Gao, Fuwei Zhang, Zhao Zhang, Xiaoshuang Min, and Fuzhen Zhuang. 2025. Mixed-curvature multi-modal knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 39, pages 11699–11707. find
  13. Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang, and Longbo Huang. 2022. Modality competition: What makes joint training of multi-modal network fail in deep learning?(provably). In International conference on machine learning (ICML), pages 9226– 9259. find
  14. Yue Jian, Xiangyu Luo, Zhifei Li, Miao Zhang, Yan Zhang, Kui Xiao, and Xiaoju Hou. 2025. Apkgc: Noise-enhanced multi-modal knowledge graph completion with attention penalty. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 39, pages 15005–15013. find
  15. Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowledge. In Proceedings of the 16th international conference on World Wide Web, pages 697–706. find
  16. Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197.
  17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems (NeurIPS), 30.
  18. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78–85.
  19. Yunpeng Wang, Bo Ning, Xin Wang, Chengfei Liu, and Guanyu Li. 2025. Segmentation similarity enhanced semantic related entity fusion for multi-modal knowledge graph completion. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR),… find
  20. Zikang Wang, Linjing Li, Qiudan Li, and Daniel Zeng. 2019. Multimodal data enhanced representation learning for knowledge graphs. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE. find
  21. Ruobing Xie, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Image-embodied knowledge representation learning. arXiv preprint arXiv:1609.07028. find
  22. Derong Xu, Tong Xu, Shiwei Wu, Jingbo Zhou, and Enhong Chen. 2022. Relation-enhanced negative sampling for multimodal knowledge graph completion. In Proceedings of the 30th ACM international conference on multimedia (MM), pages 3857–3866. find
  23. Bishan Yang, Scott Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In Proceedings of the International Conference on Learning Representations (ICLR).
  24. Yichi Zhang, Zhuo Chen, Lingbing Guo, Yajing Xu, Binbin Hu, Ziqi Liu, Wen Zhang, and Huajun Chen. 2025a. Multiple heads are better than one: Mixture of modality knowledge experts for entity representation learning. In The Thirteenth International Conference on Learning Representations (ICLR). find
  25. Yichi Zhang, Zhuo Chen, Lingbing Guo, Yajing Xu, Binbin Hu, Ziqi Liu, Wen Zhang, and Huajun Chen. 2025b. Tokenization, fusion, and augmentation: Towards fine-grained multi-modal entity representation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 39, pages 13322–133… find
  26. Yichi Zhang, Zhuo Chen, Lei Liang, Huajun Chen, and Wen Zhang. 2024. Unleashing the power of imbalanced modality information for multi-modal knowledge graph completion. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COL… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peng-2026-not-all-modalities-once-dynamic/