Elamur Research⌕
☰
home / papers / 2022 / Modern Baselines for SPARQL Semantic Parsing

Modern Baselines for SPARQL Semantic Parsing

Paper · 2022 · Debayan Banerjee, Pranav Ajit Nair, Jivat Neet Kaur et al.

Summary

The paper addresses the query building stage in KGQA systems, i.e., the conversion of a natural-language question into a SPARQL query that can be executed over a knowledge graph. The authors assume that entities and relations have already been correctly linked to the graph. They examine whether ordinary pre-trained models can replace specialized approaches based on templates or query graphs. To this end, they compare T5 (small and base), BART-base, and the LSTM network PGN with BERT and KG embeddings on the LC-QuAD 1.0 (DBpedia) and LC-QuAD 2.0 (Wikidata) datasets. The main technical device is encoding IRI prefixes and SPARQL keywords with T5 sentinel tokens. Without this, the tokenizer splits URLs apart, and accuracy drops almost to zero. With this encoding, T5 outperforms all previous systems on both datasets. Moreover, the model can transfer fragments of the question, such as numbers, directly into the query, something no previous QB approach had done. The authors argue that SPARQL parsing should rely more on pre-training than on custom architectures. The code and data are publicly available.

Key points

  • Model sizes: T5-small has 60M parameters, T5-base 220M (pre-trained on C4, 750 GB), BART-base 139M (160 GB of text). PGN has 53M parameters and was not pre-trained.
  • The LC-QuAD 2.0 test split was reduced from 6046 to 4211 questions. Only those were kept for which the gold query returns an answer on the Wikidata dump of 13.10.2021 deployed in Virtuoso.
  • On LC-QuAD 2.0 questions that require copying a fragment of the question into the query, T5 reaches an F1 of about 0.8, whereas BART reaches only about 0.3. T5-small and T5-base perform almost identically.
  • Removing the KG embeddings from PGN reduces accuracy by about 20 pp. Re-ranking hypotheses with BERT adds 10–20 pp, but PGN still falls short of T5.
  • The authors manually analyzed 100 random errors from each model. T5 and BART exhibit "copy morphing", i.e., distortion of copied IDs (for example, notableWork becomes notabilityWork). In T5 this is observed only on DBpedia, and PGN shows no such behavior. Typical errors: swapped triple elements and incorrect variables. Repository: github.com/debayan/sigir2022-sparqlbaselines.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universität HamburgHamburg · Germany84 works · 5,692 citations🏢Microsoft ResearchBengaluru · India19 works · 4,213 citations🎓Indian Institute of Technology BHUVaranasi · India11 works · 596 citations🎓Banaras Hindu UniversityVaranasi · India7 works · 308 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 2Research subjects and languages the work is about.Knowledge graphs & ontologies · 218 worksSemantic parsing & meaning representation · 194 works

Cite

Cite as Debayan Banerjee, Pranav Ajit Nair, Jivat Neet Kaur, Ricardo Usbeck and Chris Biemann. 2022. Modern Baselines for SPARQL Semantic Parsing. arXiv:2204.12793.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/3477495.3531841 · arXiv 2204.12793

@misc{banerjee2022modern,
  title = {{Modern Baselines for SPARQL Semantic Parsing}},
  author = {Debayan Banerjee, Pranav Ajit Nair, Jivat Neet Kaur, Ricardo Usbeck and Chris Biemann},
  year = {2022},
  doi = {10.1145/3477495.3531841},
  eprint = {2204.12793},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3477495.3531841}
}

References

46 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdalghani Abujabal, Mohamed Yahya, Mirek Riedewald, and Gerhard Weikum. 2017. Automated Template Generation for Question Answering over Knowledge Graphs. International World Wide Web Conferences Steering Committee. find
  2. Ram G. Athreya, Srividya Kona Bansal, Axel-Cyrille Ngonga Ngomo, and Ricardo Usbeck. 2021. Template-based Question Answering using Recursive Neural Networks. In 15th IEEE International Conference on Semantic Computing, ICSC 2021, Laguna Hills, CA, USA, January 27-29, 2021. IEEE. find
  3. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A Collaboratively Created Graph Database for Structuring Human Knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data. Association for Computing Machinery. https://do… find
  4. Antoine Bordes, Nicolas Usunier, Alberto Garcia-Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating Embeddings for Modeling Multi-Relational Data (NIPS). Curran Associates Inc.
  5. Angel X. Chang and Christopher Manning. 2012. SUTime: A library for recognizing and normalizing time expressions. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12). European Language Resources Association (ELRA). find
  6. Yongrui Chen, Huiying Li, Yuncheng Hua, and Guilin Qi. 2020. Formal Query Building with Query Structure Prediction for Complex Question Answering over Knowledge Base. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20. International Joint Conferenc… find
  7. Dennis Diefenbach, Vanessa Lopez, Kamal Singh, and Pierre Maret. 2018. Core Techniques of Question Answering Systems over Knowledge Bases: A Survey. Knowledge and Information Systems (2018). find
  8. Jiwei Ding, Wei Hu, Qixin Xu, and Yuzhong Qu. 2019. Leveraging Frequent Query Substructures to Generate Formal Queries for Complex Question Answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langu… find
  9. Mohnish Dubey, Debayan Banerjee, Abdelrahman Abdelkawi, and Jens Lehmann. 2019. LC-QuAD 2.0: A Large Dataset for Complex Question Answering over Wikidata and DBpedia. In The Semantic Web – ISWC 2019. Springer International Publishing. find
  10. Mohnish Dubey, Debayan Banerjee, Debanjan Chaudhuri, and Jens Lehmann. 2018. EARL: Joint Entity and Relation Linking for Question Answering over Knowledge Graphs. In Proceedings of the 17th International Semantic Web Conference (ISWC). find
  11. Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Schärli. 2020. Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures. https://arxiv.org/abs/2007.08970 ↗
  12. Mikhail Galkin, Priyansh Trivedi, Gaurav Maheshwari, Ricardo Usbeck, and Jens Lehmann. 2020. Message Passing for Hyper-Relational Knowledge Graphs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. ACL. find
  13. Aaron Gokaslan and Vanya Cohen. 2019. OpenWebText Corpus. http://Skylion007.github.io/OpenWebTextCorpus ↗
  14. Jonathan Herzig, Peter Shaw, Ming-Wei Chang, Kelvin Guu, Panupong Pasupat, and Yuan Zhang. 2021. Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations. arXiv preprint arXiv:2104.07478 (2021). — [arXiv:2104.07478](https://arxiv.org/abs/2104.07478) find
  15. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation (1997). find
  16. Sen Hu, Lei Zou, and Xinbo Zhang. 2018. A State-transition Framework to Answer Complex Questions over Knowledge Base. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. ACL. find
  17. Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. 2021. Complex Temporal Question Answering on Knowledge Graphs. Association for Computing Machinery. find
  18. Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2020. Measuring Compositional Generalization: A Comprehensive Method on… find
  19. Yunshi Lan and Jing Jiang. 2020. Query Graph Generation for Answering Multi-hop Complex Questions from Knowledge Bases. In Proceedings of the 58th Annual Meeting of the ACL. ACL, Online. find
  20. Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. 2015. DBpedia – A Large-Scale, Multilingual Knowledge Base Extracted from Wikipedia. Semantic Web (2015).
  21. Adam Lerer, Ledell Wu, Jiajun Shen, Timothee Lacroix, Luca Wehrstedt, Abhijit Bose, and Alex Peysakhovich. 2019. PyTorch-BigGraph: A Large-scale Graph Embedding System. In Proceedings of the 2nd SysML Conference. find
  22. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58th Annual Meeting of the…
  23. Kangqi Luo, Fengli Lin, Xusheng Luo, and Kenny Zhu. 2018. Knowledge Base Question Answering via Encoding of Complex Query Graphs. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. ACL. find
  24. Joel Mackenzie, Rodger Benham, Matthias Petri, Johanne R. Trippas, J. Shane Culpepper, and Alistair Moffat. 2020. CC-News-En: A Large English News Corpus. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (CIKM ’20). ACM. find
  25. Aleksandr Perevalov, Xi Yan, Liubov Kovriguina, Longquan Jiang, Andreas Both, and Ricardo Usbeck. 2022. Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis. CoRR abs/2201.08174 (2022). arXiv:2201.08174 https://arxiv.org/abs/2201.08174 ↗
  26. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research (2020).
  27. Subendhu Rongali, Luca Soldaini, Emilio Monti, and Wael Hamza. 2020. Don’t Parse, Generate! A Sequence to Sequence Architecture for Task-Oriented Semantic Parsing. Proceedings of The Web Conference 2020 (2020). find
  28. Ahmad Sakor, Kuldeep Singh, and Maria-Esther Vidal. 2019. FALCON: An Entity and Relation Linking Framework over DBpedia. In Proceedings of the ISWC 2019 Satellite Tracks (Posters & Demonstrations, Industry, and Outrageous Ideas) co-located with 18th International Semantic Web Conference (ISWC 2019),… find
  29. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. In Proceedings of the 55th Annual Meeting of the ACL. ACL. find
  30. Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021. Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International… find
  31. Saeedeh Shekarpour, Edgard Marx, Axel-Cyrille Ngonga Ngomo, and Sören Auer. 2015. SINA: Semantic interpretation of user queries for question answering on interlinked data. Journal of Web Semantics (2015). find
  32. Kuldeep Singh, Andreas Both, Dennis Diefenbach, Saeedeh Shekarpour, Didier Cherix, and Christoph Lange. 2016. Qanary - The Fast Track to Creating a Question Answering System with Linked Data Technology. In The Semantic Web - ESWC 2016 Satellite Events, Heraklion, Crete, Greece, May 29 - June 2, 2016… find
  33. Kuldeep Singh, Ioanna Lytra, Arun Sethupat Radhakrishna, Akhilesh Vyas, and Maria-Esther Vidal. 2018. Dynamic Composition of Question Answering Pipelines with FRANKENSTEIN. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, U… find
  34. Kuldeep Singh, Arun Sethupat Radhakrishna, Andreas Both, Saeedeh Shekarpour, Ioanna Lytra, Ricardo Usbeck, Akhilesh Vyas, Akmal Khikmatullaev, Dharmen Punjani, Christoph Lange, Maria Esther Vidal, Jens Lehmann, and Sören Auer. 2018. Why Reinvent the Wheel: Let’s Build Question Answering Systems Toge… find
  35. Tommaso Soru, Edgard Marx, Diego Moussallem, Gustavo Publio, André Valdes-tilhas, Diego Esteves, and Ciro Baron Neto. 2017. SPARQL as a Foreign Language. In Proceedings of the 13th International Conference on Semantic Systems - SEMAN-TiCS2017 Posters and Demos. find
  36. Jannik Strötgen and Michael Gertz. 2010. HeidelTime: High Quality Rule-Based Extraction and Normalization of Temporal Expressions. In Proceedings of the 5th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  37. Yawei Sun, Lingling Zhang, Gong Cheng, and Yuzhong Qu. 2020. SPARQA: Skeleton-Based Semantic Parsing for Complex Questions over Knowledge Bases. Proceedings of the AAAI Conference on Artificial Intelligence 34 (2020). find
  38. Trieu H. Trinh and Quoc V. Le. 2018. A Simple Method for Commonsense Reasoning. CoRR abs/1806.02847 (2018). arXiv:1806.02847 http://arxiv.org/abs/ 1806.02847 ↗
  39. Priyansh Trivedi, Gaurav Maheshwari, Mohnish Dubey, and Jens Lehmann. 2017. LC-QuAD: A Corpus for Complex Question Answering over Knowledge Graphs. In The Semantic Web – ISWC 2017. Springer International Publishing. find
  40. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2017/file/ 3f5ee243547dee9… ↗
  41. Daniel Vollmers, Rricha Jalota, Diego Moussallem, Hardik Topiwala, Axel-Cyrille Ngonga Ngomo, and Ricardo Usbeck. 2021. Knowledge Graph Question Answering Using Graph-Pattern Isomorphism. (2021). find
  42. Vrandečić, Denny and Krötzsch, Markus. 2014. Wikidata: A Free Collaborative Knowledge Base. Commun. ACM (2014). find
  43. Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic Parsing via Staged Query Graph Generation: Question Answering with Knowledge Base. In Proceedings of the 53rd Annual Meeting of the ACL and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long…
  44. Hamid Zafar, Giulio Napolitano, and Jens Lehmann. 2018. Formal Query Generation for Question Answering over Knowledge Bases. In The Semantic Web. Springer International Publishing. find
  45. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. In The IEEE International Conference on Computer Vision (ICCV).
  46. Jianyun Zou, Min Yang, Lichao Zhang, Yechen Xu, Qifan Pan, Fengqing Jiang, Ran Qin, Shushu Wang, Yifan He, Songfang Huang, and Zhou Zhao. 2021. A Chinese Multi-type Complex Questions Answering Dataset over Wikidata. arXiv:2111.06086 [cs.CL] — [arXiv:2111.06086](https://arxiv.org/abs/2111.06086) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/banerjee-2022-modern-baselines-sparql-semantic-parsing/