Elamur Research⌕
☰
home / papers / 2026 / How Do Inpainting Artifacts Propagate to Language?

How Do Inpainting Artifacts Propagate to Language?

Paper · 2026 · Pratham Yashwante, Davit Abrahamyan, Shresth Grover et al.

Summary

When part of an image is reconstructed by diffusion inpainting before captioning, the captioning model cannot tell which pixels are synthetic. Invented content can therefore leak into the generated text. The authors build a diagnostic pipeline with three steps:

  1. Degrade a region of the image with a hard center mask, a Gaussian-blurred mask, or low-dimensional compression.
  2. Fill the region with Stable Diffusion 1.5, 2.0 or 3.0.
  3. Caption both the original and the reconstructed version with frozen VLMs (LLaVA, BLIP, Qwen2.5-VL).

Captions are compared with lexical metrics (BLEU, METEOR, ROUGE-L) and semantic metrics (SimCSE, SBERT). A frozen ViT-B encoder is also probed for changes in embeddings and attention. Experiments span natural photos, medical scans, audio spectrograms and time-series plots.

Higher reconstruction fidelity, particularly as measured by LPIPS and MSE, goes together with better captions, while SSIM predicts caption quality poorly. Hard center masks cause the most damage. Attention drift concentrates in deeper encoder layers and inside the inpainted area. The authors conclude that systems feeding edited images into VLMs should treat reconstruction quality as a factor in output reliability.

Key points

  • Datasets: Flickr, RefCOCOg, ROCOv2, Indiana X-Ray, GTZAN and TRUCE.
    • Fidelity–caption correlations are strongest on RefCOCOg and somewhat weaker on TRUCE.
    • GTZAN and X-ray give no meaningful correlation because their captions vary little.
  • Entity overlap on Flickr (SD1.5, spaCy pipeline):
    • Nouns: precision 0.58, recall 0.32, F1 0.40.
    • Verbs: F1 0.33.
    • Adjectives: F1 0.16.
  • Manual review of the 100 worst Flickr cases:
    • Attribute detail loss: 98%.
    • Participant reassignment: 85%.
    • Object substitution: 29%.
    • Activity substitution: 22%.
  • Attention drift in ViT-B (Total Variation Distance between CLS-to-patch attention maps):
    • Drift grows with depth and is largest for center masks.
    • Drift is higher inside the reconstructed region than outside it.
    • Late-layer drift correlates negatively with entity F1 (r = −0.51, p < 0.001).
  • Robustness and resources:
    • SAM object masks confirm the fidelity–caption trend.
    • UMAP plots of CLS embeddings show center-masked images drifting away from the originals.
    • Code is available at github.com/raosukruth/inpaint-caption (ACL 2026, short paper).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California San DiegoSan Diego · United States89 works · 54,032 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Pratham Yashwante, Davit Abrahamyan, Shresth Grover and Sukruth Rao. 2026. How Do Inpainting Artifacts Propagate to Language?. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-short.60

@inproceedings{yashwante2026how,
  title = {{How Do Inpainting Artifacts Propagate to Language?}},
  author = {Pratham Yashwante, Davit Abrahamyan, Shresth Grover and Sukruth Rao},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-short.60},
  url = {https://doi.org/10.18653/v1/2026.acl-short.60}
}

References

31 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077–6086. find
  2. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.
  3. Dina Demner-Fushman, Marc D. Kohli, Marc B. Rosenman, Sonya E. Shooshan, Laritza Rodriguez, Sameer Antani, George R. Thoma, and Clement J. McDonald. 2016. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):3… find
  4. SeungHeon Doh, Keunwoo Choi, Jongpil Lee, and Juhan Nam. 2023. Lp-musiccaps: Llm-based pseudo music captioning. arXiv preprint arXiv:2307.16372. — [arXiv:2307.16372](https://arxiv.org/abs/2307.16372) find
  5. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. Preprint,…
  6. Seth* Forsgren and Hayk* Martiros. 2022. Riffusion -Stable diffusion for real-time music generation. find
  7. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 6894–6910.
  8. Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261.
  9. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spacy: Industrial-strength natural language processing in python. find
  10. Harsh Jhamtani and Taylor Berg-Kirkpatrick. 2021. Truth-conditional captioning of time series data. arXiv preprint arXiv:2110.01839. — [arXiv:2110.01839](https://arxiv.org/abs/2110.01839) find
  11. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023. Segment anything. Preprint, arXiv:2304.02643.
  12. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024. Llava-onevision: Easy visual task transfer. Preprint, arXiv:2408.03326.
  13. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR.
  14. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  15. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695.
  16. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer.
  17. Johannes Rückert, Louise Bloch, Raphael Brüngel, Ahmad Idrissi-Yaghir, Henning Schäfer, Cynthia S. Schmidt, Sven Koitka, Obioma Pelka, Asma Ben Abacha, Alba G. Seco de Herrera, Henning Müller, Peter A. Horn, Felix Nensa, and Christoph M. Friedrich. 2024. Rocov2: Radiology objects in context version… find
  18. Zachary Shah, Neelesh Ramachandran, and Mason Wang. 2024. Riff-controlnet: Controlled audio inpainting using controlnet architecture. https://github.com/zachary-shah/riff-cnet. ↗
  19. Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253.
  20. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  21. Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. 2016. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544. find
  22. Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641–2649… find
  23. Qwen Team. 2024. Qwen2.5-vl: A family of multimodal large language models. https://qwenlm.github.io/blog/qwen2.5-vl/. ↗
  24. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 3982–3992.
  25. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035–4045.
  26. George Tzanetakis and Perry Cook. 2002. Musical genre classification of audio signals. IEEE Transactions on speech and audio processing, 10(5):293– 302. find
  27. Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612. find
  28. Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2023. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (IC… find
  29. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048–2057. PMLR.
  30. Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. 2016. Modeling context in referring expressions. In European conference on computer vision, pages 69–85. Springer.
  31. Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yashwante-2026-how-do-inpainting-artifacts-propagate/