Elamur Research⌕
☰
home / papers / 2026 / HotelCheckSpan: A Benchmark Dataset for LLM Faithfulness

HotelCheckSpan: A Benchmark Dataset for LLM Faithfulness

Summary

LLM-generated summaries often contain hallucinations. These are especially hard to judge in hotel marketing text, where verifiable facts sit next to subjective praise, so the boundary between a real error and acceptable embellishment is blurry. The authors build HotelCheckSpan, which they present as the first span-level faithfulness benchmark for this domain. They gathered English accommodation descriptions from online travel agencies and hotel websites, merged and de-duplicated the texts available for each property, and had Gemini 1.5 Flash write short, topic-focused highlights. These outputs were left unedited so that natural model errors were preserved. Human annotators then marked the minimal faulty fragments in each summary and labelled them as Incorrect, Misleading, or Not Checkable. Several LLMs performed the same task, which allows direct comparison with people. The analysis shows that agreement looks different when measured per example versus per span. Soft and hard F1 variants separate disputes about span boundaries from disputes about labels. LLM judges proved unevenly reliable. The work argues for reporting inter-annotator agreement at multiple granularities.

Key points

  • The dataset contains 496 description–summary pairs, divided into equal "shorter" and "longer" splits. Inputs average 716.5 words (range 20–1,517), while summaries average 14.7 words, typically one sentence.
  • The properties come from 44 countries, led by the US (about 30%), the UK (8%) and Japan (7%), and about 80% are standard hotels. Duplicate segments were removed when BLEU or ROUGE-L F1 exceeded 0.8.
  • Summaries were produced with Gemini 1.5 Flash, following the generation approach of Kamath et al. (2024), which used PaLM 2.
  • When GPT-4o was used as an annotator, it marked too many spans. A "less strict" prompt meant to correct this actually led to even more spans being flagged.
  • The unannotated pairs, guidelines and code are public on GitHub and Hugging Face. The human and LLM annotations are available only on request, to avoid test-set contamination. Everything is licensed under CC BY-NC 4.0.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3x4p5r52zmvw

@inproceedings{schmidtova2026hotelcheckspan,
  title = {{HotelCheckSpan: A Benchmark Dataset for LLM Faithfulness}},
  author = {Schmidtova, Patricia and Dusek, Ondrej and Mahamood, Saad},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3x4p5r52zmvw},
  url = {https://doi.org/10.63317/3x4p5r52zmvw}
}

References

28 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondřej Dušek. 2024. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics. Association for Comp… find our card
  2. Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav 8 https://researcher-help.prolific.com/en/article/2273bd 9 https://www.livingwage.org.uk/ ↗
  3. Nakov. 2019. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5636–5646, Hong Kong, China. Association for Com… find
  4. Noam Dahan and Gabriel Stanovsky. 2025. The state and fate of summarization datasets: A survey. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7259–7278, Albuque… find
  5. Gemini Team. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. find our card
  6. Gemma Team. 2025. Gemma 3 technical report. find
  7. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38. find our card
  8. Zdeněk Kasner, Ondrej Platek, Patricia Schmidtova, Simone Balloccu, and Ondrej Dusek. 2024. factgenie: A framework for span-based evaluation of generated texts. In Proceedings of the 17th International Natural Language Generation Conference: System Demonstrations, pages 13– 15, Tokyo, Japan. Associa… find
  9. Zdeněk Kasner, Vilém Zouhar, Patrícia Schmidtová, Ivan Kartáč, Kristýna Onderková, Ondřej Plátek, Dimitra Gkatzia, Saad Mahamood, Ondřej Dušek, and Simone Balloccu. 2025. Large language models as span annotators. find
  10. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics. find
  11. OpenAI Team. 2024. Gpt-4o system card. find our card
  12. Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag… find our card
  13. Barbara Plank. 2022. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find our card
  14. Patrícia Schmidtová, Ondrej Dusek, and Saad Mahamood. 2025. Real-world summarization: When evaluation reaches its limits. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 25014–25026, Suzhou, China. Association for Computational Linguistics. find
  15. PaLM 2 Team. 2023. Palm 2 technical report. find
  16. Craig Thomson and Ehud Reiter. 2020a. A gold standard methodology for evaluating accuracy in data-to-text systems. In Proceedings of the 13th International Conference on Natural Language Generation, INLG 2020, pages 158–168, Dublin, Ireland. find
  17. Craig Thomson and Ehud Reiter. 2020b. A gold standard methodology for evaluating accuracy in data-to-text systems. In Proceedings of the 13th International Conference on Natural Language Generation, pages 158–168, Dublin, Ireland. Association for Computational Linguistics. find
  18. Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation, pages 355–368, Tokyo, Japan. Association for Comp… find
  19. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. 12. find
  20. Language Resource References Md. Hijbul Alam, Woo-Jong Ryu, and SangKeun Lee. 2016. Joint multi-grain topic sentiment: modeling semantic aspects for online reviews. Information Sciences, 339:206–223. find
  21. Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. 2021. Aspect-controllable opinion summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6578–6593, Online and Punta Cana, Dominican Republic. Association for Computational Linguisti… find
  22. Antognini, Diego and Faltings, Boi. 2020. HotelRec: a Novel Very Large-Scale Hotel Recommendation Dataset. European Language Resources Association. find
  23. Forrest Sheng Bao, Miaoran Li, Renyi Qu, Ge Luo, Erana Wan, Yujia Tang, Weisi Fan, Manveer Singh Tamber, Suleman Kazi, Vivek Sourabh, Mike Qi, Ruixuan Tu, Chenyu Xu, Matthew Gonzales, Ofer Mendelevitch, and Amin Ahmad. 2025. FaithBench: A diverse hallucination benchmark for summarization by Modern L… find
  24. Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. find
  25. Srinivas Ramesh Kamath, Fahime Same, and Saad Mahamood. 2024. Generating hotel highlights from unstructured text using LLMs. In Proceedings of the 17th International Natural Language Generation Conference, pages 280–288, Tokyo, Japan. Association for Computational Linguistics. find
  26. Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332–9346, Online. Association for Computational… find
  27. Mir Tafseer Nayeem and Davood Rafiei. 2024. Lfosum: Summarizing long-form opinions with large language models. find
  28. Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, and Greg Durrett. 2023. Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors. In Proceedings of the 61st Annual Meeting of the Association fo… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schmidtova-2026-hotelcheckspan-benchmark-dataset-llm-faithfulness/