Elamur Research⌕
☰
home / papers / 2026 / Knowledge Extraction on Semi-Structured Content: Does It Remain Relevant for Question Answering in the Era of LLMs?

Knowledge Extraction on Semi-Structured Content: Does It Remain Relevant for Question Answering in the Era of LLMs?

Paper · 2026 · Kai Sun, Yin Huang, Srishti Mehra et al.

Summary

LLMs now answer questions over semi-structured web content (tables, attribute–value lists, free-form layouts) quite well. The authors ask whether extracting (subject, predicate, object) triples is still useful for QA. To find out, they extend the WebSRC benchmark in two ways. They add manually annotated triples for sampled cleaned pages, and they add a new set of raw, uncleaned webpages from popular Common Crawl domains that better reflect what a RAG system actually retrieves.

They then test commercial and open-source LLMs of different sizes on three things: extraction quality, QA with triple augmentation, and multi-task fine-tuning. They also try a cheaper option in which an LLM writes extraction scripts. Extraction is solid on cleaned pages but weak on whole pages. Even so, triples still raise QA accuracy, whether they are supplied at inference time or taught through multi-task training, and the two approaches add up. The benefit is largest for small models and noisy pages. This suggests knowledge extraction still has a place in LLM-based web QA.

Key points

  • Dataset (released as SemiBench, non-commercial):
    • Cleaned part: 180 pages from 60 WebSRC websites, 6,186 QA pairs, 4,741 triples.
    • Whole-page part: 251 pages from 139 sites, 1,006 QA pairs, 30,263 triples (about 121 per page).
  • Triple extraction (F1):
    • LLMs reach about 77% on cleaned pages, versus 46% for the earlier GNN-based OpenIE method.
    • On whole pages this falls to roughly 28%.
  • QA accuracy:
    • About 95% on cleaned pages but only 76% on whole pages.
    • Adding ground-truth triples brings whole pages to 84%.
    • Multi-task fine-tuning (QA plus extraction) reaches 83.5%.
    • Combining both reaches about 87%.
  • Script-based extraction:
    • LLM-written scripts achieve 61% F1 on cleaned pages, comparable to fine-tuned Llama models (51% for 3B, 72% for 70B).
    • The resulting triples give roughly a 5% QA gain over zero-shot.
    • The approach has not yet worked on whole pages.
  • Practical guidance:
    • 70B-class models extract reasonably well with prompting alone; Trafilatura cleaning helps when pages are too long for the context window.
    • 3B models are too weak for high-quality extraction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksQuestion answering · 441 works

Cite

Cite as Kai Sun, Yin Huang, Srishti Mehra, Mohammad Kachuee, Xilun Chen, Renjie Tao, Zhaojiang Lin, Andrea Jessee, Nirav Shah, Alex L Betty, Yue Liu, Anuj Kumar, Wen-tau Yih and Xin Luna Dong. 2026. Knowledge Extraction on Semi-Structured Content: Does It Remain Relevant for Question Answering in the Era of LLMs?. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.91

@inproceedings{sun2026knowledge,
  title = {{Knowledge Extraction on Semi-Structured Content: Does It Remain Relevant for Question Answering in the Era of LLMs?}},
  author = {Kai Sun, Yin Huang, Srishti Mehra, Mohammad Kachuee, Xilun Chen, Renjie Tao, Zhaojiang Lin, Andrea Jessee, Nirav Shah, Alex L Betty, Yue Liu, Anuj Kumar, Wen-tau Yih and Xin Luna Dong},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.91},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.91}
}

References

40 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, and 110 others. 2024. Phi-… — Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
  2. Aaron Adcock, Aayushi Srivastava, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pande, Abhinav Pandey, Abhinav Sharma, Abhishek Kadian, Abhishek Ku-mawat, Adam Kelsey, and 1 others. 2026. The llama 4 herd: Architecture, training, evaluation, and deployment notes. arXiv preprint arXiv:2601.11659. — [arXiv:2601.11659](https://arxiv.org/abs/2601.11659) find
  3. Adrien Barbaresi. 2021. Trafilatura: A web scraping library and command-line tool for text discovery and extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstra…
  4. Mirko Bronzi, Valter Crescenzi, Paolo Merialdo, and Paolo Papotti. 2013. Extraction and integration of partially overlapping web sources. Proc. VLDB Endow., 6(10):805–816. find
  5. Pere-Lluís Huguet Cabot and Roberto Navigli. 2021. Rebel: Relation extraction by end-to-end language generation. In Findings of the association for computational linguistics: emnlp 2021, pages 2370–2381. find
  6. Michael Cafarella, Alon Halevy, Hongrae Lee, Jayant Madhavan, Cong Yu, Daisy Zhe Wang, and Eugene Wu. 2018. Ten years of webtables. Proc. VLDB Endow., 11(12):2140–2149. find
  7. Xingyu Chen, Zihan Zhao, Lu Chen, JiaBao Ji, Danyang Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. 2021. WebSRC: A dataset for web-based structural reading comprehension. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4173–4185, Online and Punta Cana, Dom… find
  8. Michael Han Daniel Han and Unsloth team. 2023. Unsloth. find
  9. Xiang Deng, Prashant Shiralkar, Colin Lockard, Binxuan Huang, and Huan Sun. 2022. Dom-lm: Learning generalizable representations for html documents. Preprint, arXiv:2201.10608. — [arXiv:2201.10608](https://arxiv.org/abs/2201.10608) find
  10. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Preprint, arXiv:2305.14314.
  11. Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. 2014. Knowledge vault: a web-scale approach to probabilistic knowledge fusion. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data M… find
  12. Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information extraction. In Proceedings of the 2011 conference on empirical methods in natural language processing, pages 1535–1545. find
  13. Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. 2025. Colpali: Efficient document retrieval with vision language models. Preprint, arXiv:2407.01449. find
  14. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  15. P. Gulhane, A. Madaan, R. R. Mehta, J. Ramamirtham, R. Rastogi, S. Satpal, S. H. Sengamedu, A. Tengli, and C. Tiwari. 2011. Web-scale information extraction with vertex. In ICDE. find
  16. Qiang Hao, Rui Cai, Yanwei Pang, and Lei Zhang. 2011. From one tree to a forest: a unified solution for structured web data extraction. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’11, page 775–784, New York, NY, USA. Asso… find
  17. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. WebVoyager: Building an end-to-end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p…
  18. Zhi Hong, Kyle Chard, and Ian Foster. 2024. Combining language and graph models for semi-structured information extraction on the web. Preprint, arXiv:2402.14129. — [arXiv:2402.14129](https://arxiv.org/abs/2402.14129) find
  19. Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Xin Zhao, and Ji-Rong Wen. 2023. StructGPT: A general framework for large language model to reason over structured data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9237–9251, Singapore. Association f… find
  20. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2017. Answering complex questions using open information extraction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 311–316, Vancouver, Canada. Association for Computational Li…
  21. N. Kushmerick, D. S. Weld, and R. B. Doorenbos. 1997. Wrapper induction for information extraction. In IJCAI. find
  22. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–…
  23. Shicheng Liu, Kai Sun, Lisheng Fu, Xilun Chen, Xinyuan Zhang, Zhaojiang Lin, Rulin Shao, Yue Liu, Anuj Kumar, Wen tau Yih, and Xin Luna Dong. 2025. Scribes: Web-scale script-based semi-structured data extraction with reinforcement learning. Preprint, arXiv:2510.01832. — [arXiv:2510.01832](https://arxiv.org/abs/2510.01832) find
  24. Colin Lockard, Xin Luna Dong, Arash Einolghozati, and Prashant Shiralkar. 2018. Ceres: distantly supervised relation extraction from the semi-structured web. Proc. VLDB Endow., 11(10):1084–1096. find
  25. Colin Lockard, Prashant Shiralkar, and Xin Luna Dong. 2019. OpenCeres: When open information extraction meets the semi-structured web. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and…
  26. Colin Lockard, Prashant Shiralkar, Xin Luna Dong, and Hannaneh Hajishirzi. 2020. ZeroShotCeres: Zero-shot relation extraction from semi-structured webpages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8105–8117, Online. Association for Computatio… find
  27. Mausam Mausam. 2016. Open information extraction systems and downstream applications. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI’16, page 4074–4077. AAAI Press. find
  28. James Munkres. 1957. Algorithms for the assignment and transportation problems. Journal of the society for industrial and applied mathematics, 5(1):32–38. find
  29. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. Gpt-4o system card.…
  30. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  31. Ritesh Sarkhel, Binxuan Huang, Colin Lockard, and Prashant Shiralkar. 2023. Self-training for label-efficient information extraction from semi-structured web-pages. Proc. VLDB Endow., 16(11):3098–3110. find
  32. Rohini Srihari, Wei Li, and X Li. 1999. Information extraction supported question answering. In TREC. find
  33. Jiejun Tan, Zhicheng Dou, Wen Wang, Mang Wang, Weipeng Chen, and Ji-Rong Wen. 2025. Htmlrag: Html is better than plain text for modeling retrieved knowledge in rag systems. In Proceedings of the ACM on Web Conference 2025, WWW ’25, page 1733–1746. ACM. find
  34. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  35. Qifan Wang, Yi Fang, Anirudh Ravula, Fuli Feng, Xiaojun Quan, and Dongfang Liu. 2022. Webformer: The web-page transformer for structure information extraction. Preprint, arXiv:2202.00217. — [arXiv:2202.00217](https://arxiv.org/abs/2202.00217) find
  36. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transf…
  37. Chenhao Xie, Wenhao Huang, Jiaqing Liang, Chengsong Huang, and Yanghua Xiao. 2021. Webke: Knowledge extraction from semi-structured web with pre-trained markup language model. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, CIKM ’21, page 2211–2220, New… find
  38. Ge Zhang, Jian Wang, and Jing Ling Wang. 2024. Joint information extraction from semi-structured web pages using BERT. In Fourth International Conference on Advanced Algorithms and Neural Networks (AANN 2024), volume 13416, page 134161X. International Society for Optics and Photonics, SPIE. find
  39. Zihan Zhao, Lu Chen, Ruisheng Cao, Hongshen Xu, Xingyu Chen, and Kai Yu. 2022. TIE: Topological information enhanced structural reading comprehension on web pages. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tec… find
  40. Xiaohan Zou. 2020. A survey on application of knowledge graph. Journal of Physics: Conference Series, 1487(1):012016. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2026-knowledge-extraction-semi-structured-content/