Elamur Research⌕
☰
home / papers / 2026 / VLCE: A Knowledge-Enhanced Framework for Image Description in Disaster Assessment

VLCE: A Knowledge-Enhanced Framework for Image Description in Disaster Assessment

Paper · 2026 · Md. Mahfuzur Rahman, Marufa Kamal, Fahad Rahman et al.

Summary

General-purpose VLMs such as LLaVA and QwenVL describe post-disaster satellite and drone imagery in vague, everyday language. They miss the damage-specific detail that emergency teams need. VLCE addresses this with a two-step pipeline. First, YOLOv8 detections are converted into a prompt, and a baseline VLM drafts a caption from it. Second, a trained sequential decoder rewrites that draft using a domain lexicon. The decoder is either a CNN-LSTM or a hierarchical cross-modal Transformer.

The lexicon is built in three steps. RAKE keywords are pulled from the training captions, then expanded with ConceptNet relations and WordNet synonyms. Frozen ConceptNet Numberbatch vectors serve as the embeddings. DistilBERT embeddings provide the no-KG ablation. Visual features come from domain-matched encoders: ResNet50-EuroSAT for satellite images and a ViT UAV classifier for drone images.

Evaluation uses xBD and RescueNet, scoring per-image preference against the baselines with CLIPScore and InfoMetIC. The Transformer with knowledge graph wins on most RescueNet images. Without the KG, UAV results collapse almost entirely. The work suggests that structured external knowledge is an inexpensive way to adapt captioning to specialized, high-stakes domains.

Key points

  • The enriched vocabulary has 3,195 tokens, of which 1,566 (49%) come only from ConceptNet/WordNet. Up to 10 ConceptNet terms are retrieved per keyword, and sequences are padded to length 192.
  • Benchmarks: xBD (6,369 satellite images, 3 damage classes) and RescueNet (4,494 UAV images, 12 damage classes). CNN-LSTM variants are compared with LLaVA-7B v1.5, and Transformer variants with QwenVL-7B Instruct.
  • Best result: Transformer + KG on RescueNet is preferred over QwenVL on 95.33% of images by InfoMetIC and 73.64% by CLIPScore. Without the KG, preference drops below 2% (0.08% and 0.22%).
  • On xBD the KG helps less. The CNN-LSTM without KG even scores slightly higher on CLIPScore (55.34% vs 51.10%). The authors attribute this to the more uniform overhead imagery.
  • VLCE captions mention all 272 UAV and 640 satellite reference objects, while baseline VLMs cover about 55–65%. Qualitatively, captions without the KG show hallucinations, repetition, and incoherence.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksEngineering · 133 worksMedia Technology · 34 works

Cite

Cite as Md. Mahfuzur Rahman, Marufa Kamal, Fahad Rahman, Sunzida Siddique, Ahmed Rafi Hasan, Mohd Ariful Haque, Kishor Datta Gupta and Roy George. 2026. VLCE: A Knowledge-Enhanced Framework for Image Description in Disaster Assessment. Proceedings of the 4th Workshop on Advances in Language and Vision Research (ALVR).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.alvr-main.15

@inproceedings{rahman2026vlce,
  title = {{VLCE: A Knowledge-Enhanced Framework for Image Description in Disaster Assessment}},
  author = {Md. Mahfuzur Rahman, Marufa Kamal, Fahad Rahman, Sunzida Siddique, Ahmed Rafi Hasan, Mohd Ariful Haque, Kishor Datta Gupta and Roy George},
  booktitle = {Proceedings of the 4th Workshop on Advances in Language and Vision Research (ALVR)},
  year = {2026},
  doi = {10.18653/v1/2026.alvr-main.15},
  url = {https://doi.org/10.18653/v1/2026.alvr-main.15}
}

References

25 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. O. Abbas and J. Dang. 2023. Using image captioning for automatic post-disaster damage detection. Intelligence, Informatics and Infrastructure, 4(2):66–74. find
  2. I. Alisjahbana and 1 others. 2024. DeepDama-geNet: A two-step deep-learning model for multi-disaster building damage segmentation. Preprint, arXiv:2405.04800. — [arXiv:2405.04800](https://arxiv.org/abs/2405.04800) find
  3. J. Bai and 1 others. 2023. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. Preprint, arXiv:2308.12966.
  4. L. Chen and K. Li. 2025. Multi-modal graph aggregation transformer for image captioning. Neural Networks, 181:106813. find
  5. P.-J. Chun, T. Yamane, and Y. Maemura. 2022. A deep learning-based image captioning method for bridge damage. Computer-Aided Civil and Infrastructure Engineering, 37(11):1387–1401. find
  6. Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2020. Meshed-memory transformer for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10578–10587. find
  7. S. Elbedwehy and 1 others. 2023. Enhanced image captioning using features concatenation and efficient pre-trained word embedding. Computer Systems Science and Engineering, 46:3637–3652. find
  8. Ritwik Gupta and 1 others. 2019. xBD: A dataset for assessing building damage from satellite imagery. Preprint, arXiv:1911.09296. — [arXiv:1911.09296](https://arxiv.org/abs/1911.09296) find
  9. C. Schön, S. Siebert, and F. Stolzenburg. 2019. Using ConceptNet to teach common sense to an automated theorem prover. Electronic Proceedings in Theoretical Computer Science, 311:19–24. find
  10. M. Talib, A. H. Y. Al-Noori, and J. Suad. 2024. YOLOv8-CAB: Improved YOLOv8 for real-time object detection. Karbala International Journal of Modern Science, 10(1):5. find
  11. J. Tang, P. Li, and M. Jiang. 2022. Image caption generation based on knowledge graph guidance and self-attention. Preprint. find
  12. M. Toshevska, F. Stojanovska, and J. Kalajdjieski. 2020. Comparative analysis of word embeddings for capturing word similarities. Preprint, arXiv:2005.03812. — [arXiv:2005.03812](https://arxiv.org/abs/2005.03812) find
  13. G. Wang and 1 others. 2023. UAV-YOLOv8: A small-object-detection model for UAV aerial photography. Sensors, 23(16):7190. find
  14. X. Wang and 1 others. 2024. Knowledge enhancement strategies for remote sensing image captioning using CLIP and LLMs. In Proceedings of MIDA, pages 313–318. find
  15. X. Zhai and 1 others. 2023. YOLO-Drone: An optimized YOLOv8 for tiny UAV object detection. Electronics, 12(17):3664. find
  16. Wentian Zhao and Xinxiao Wu. 2024. Boosting entity-aware image captioning with multi-modal knowledge graph. IEEE Transactions on Multimedia, 26:2659– 2670. find
  17. J. Hessel and 1 others. 2021. CLIPScore: A reference-free evaluation metric for image captioning. Preprint, arXiv:2104.08718.
  18. Y. Zhou, Y. Sun, and V. Honavar. 2019. Improving image captioning by leveraging knowledge graphs. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision, pages 539–548. find
  19. A. Hu and 1 others. 2023. InfoMetIC: An informative metric for reference-free image caption evaluation. Preprint, arXiv:2305.06002. find
  20. Z. Zou and 1 others. 2021. Disaster image classification by fusing multimodal social media data. ISPRS International Journal of Geo-Information, 10(10):636. find
  21. S. Kota and 1 others. 2022. Multimodal combination of text and image tweets for disaster response assessment. In D2R2. find
  22. H. Liu and 1 others. 2023. Visual instruction tuning. In Advances in Neural Information Processing Systems, volume 36, pages 34892–34916.
  23. Rebecca Mason and Eugene Charniak. 2014. Domain-specific image captioning. In Proceedings of the Conference on Computational Natural Language Learning, pages 28–36. Association for Computational Linguistics. find
  24. M. Rahnemoonfar, T. Chowdhury, and R. Murphy. 2023. RescueNet image classification dataset. Figshare Dataset. find
  25. A. F. Rasheed and M. Zarkoosh. 2025. Optimized YOLOv8 for multi-scale object detection. Journal of Real-Time Image Processing, 22(1):6. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rahman-2026-vlce-knowledge-enhanced-framework-image/