Elamur Research⌕
☰
home / papers / 2019 / Querying Knowledge via Multi-Hop English Questions

Querying Knowledge via Multi-Hop English Questions

Paper · 2019 · Tiantian Gao, Paul Fodor and Michael Kifer

Summary

KALM-QA extends the KALM knowledge-authoring framework so that it can answer complex multi-hop English questions over a knowledge base. A multi-hop question requires joining several relations, possibly a relation with itself. Plain controlled natural languages such as ACE give different logical forms to sentences that mean the same thing. KALM fixes this with a semantic layer on top of ACE's parser, APE. The parser's DRS output is mapped onto FrameOnt, a formalized version of FrameNet frames. Role fillers are disambiguated against BabelNet synsets, so that equivalent sentences receive one unique logical representation (ULR), which is then run as a Prolog query. The paper describes how the frame-semantic parser is built incrementally through structure learning. MetaQA questions are rewritten with the Stanford Parser so that they fit ACE syntax. On MetaQA's movie questions (about 29,000 test, over 260,000 training), KALM-QA answers 100% correctly. The machine-learning system VRN scores far lower. The authors also show that many of MetaQA's own gold answers are wrong. The work argues that logic-based pipelines with explicit linguistic knowledge can reach the precision that knowledge representation needs.

Key points

  • The pipeline runs in this order: APE syntactic parse to DRS, frame-based parsing with logical valence patterns (lvps), role-filler disambiguation over BabelNet, then construction of logic facts or queries.
  • On MetaQA's 2-hop and 3-hop questions, KALM-QA reaches 100% accuracy. VRN (Zhang et al. 2018b) falls well short.
  • For MetaQA, KALM-QA had to learn 88 lvps. The authors estimate that capturing the same semantics in bare Attempto or SBVR would take 974 hand-written background bridge rules.
  • MetaQA mislabels a large share of answers, mainly because distinct movies can share a title. As a result, end-to-end learning learned wrong answers for more than half of the 3-hop queries.
  • Validation used standardized query templates (44 + 59, with several English variants each) and a few hundred manual checks. An error report is published at github.com/tiantiangao7/kalm-qa.

Interesting ideas

  • I1. An lvp ties a trigger word to a frame and gives each role its own small grammatical-path extraction rule, such as verb->subject; a pattern applies only if every required role can be filled.
  • I2. Role fillers are stored as synsets rather than words, and some roles carry type constraints, such as the Release Year role requiring a valid year.
  • I3. Wrong frame readings are pruned by checking whether a filler can fit the role's type: "directs a film" is not about running a corporation, because a film is not a corporation.
  • I4. To verify tens of thousands of translations cheaply, constants are masked (for example, 'Bright Star' becomes xxxx), outputs are grouped by template, and only one representative per template is checked by hand.
  • I5. The authors audited the benchmark's gold answers with a script that explains each mismatch, such as distinct films sharing a title being conflated.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Frames could be triggered from UD trees as lemma plus dependency-path rules, one per VerbNet role, so every extracted role traces back to an explicit path.
  • I2 (world): Entities and states in the world model could be keyed to WordNet/Wikidata senses, with typed slots that reject values that cannot fit.
  • I3 (frames): Ambiguous verb frames could be resolved by checking argument compatibility against the ontology, and the failed check could be logged as the explanation for the choice.
  • I4 (learn): The same trick could audit silver frame or query outputs at scale by reviewing per-template exemplars instead of every sentence.
  • I5 (answer): QA on tales and school tests should keep entity IDs separate from surface names, and gold labels should be checked automatically before disagreements are counted as Mova errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stony Brook UniversityStony Brook · United States27 works · 2,057 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.English · 1,373 works

Cite

Cite as Tiantian Gao, Paul Fodor and Michael Kifer. 2019. Querying Knowledge via Multi-Hop English Questions. arXiv:1907.08176 · Theory and Practice of Logic Programming 19 (2019) 636-653.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1017/s1471068419000103 · arXiv 1907.08176

@misc{gao2019querying,
  title = {{Querying Knowledge via Multi-Hop English Questions}},
  author = {Tiantian Gao, Paul Fodor and Michael Kifer},
  year = {2019},
  doi = {10.1017/s1471068419000103},
  eprint = {1907.08176},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1017/s1471068419000103}
}

References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tiantian Gao, Paul Fodor, Michael Kifer J OHNSON , C. R., F ILLMORE , C. J., P ETRUCK , M. R., B AKER , C. F., E LLSWORTH , M. J., R UPPEN - HOFER , J., AND W OOD , E. J. 2002. FrameNet: Theory and Practice. find
  2. K UHN , T. 2014. A survey and classification of controlled natural languages. Comp. Linguistics 40, 1, 121–170. find
  3. L I , F. AND J AGADISH , H. V. 2014. Constructing an interactive natural language interface for relational databases. PVLDB 8, 1, 73–84. find
  4. L ÓPEZ , V., F ERN ÁNDEZ , M., M OTTA , E., AND S TIELER , N. 2012. PowerAqua: Supporting users in querying and exploring the semantic web. Semantic Web 3, 3, 249–265. find
  5. M ANNING , C. D., S URDEANU , M., B AUER , J., F INKEL , J., B ETHARD , S. J., AND M C C LOSKY , D. 2014. The Stanford CoreNLP natural language processing toolkit.
  6. M AUSAM , S CHMITZ , M., S ODERLAND , S., B ART , R., AND E TZIONI , O. 2012. Open language learning for information extraction. In The Joint Conf. on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL. Association for Computational Linguistics, Jeju Island, Korea, 523–534. find
  7. M C I LRAITH , S. A. AND W EINBERGER , K. Q., Eds. 2018. 32nd AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018. AAAI Press. find
  8. M ILLER , A. H., F ISCH , A., D ODGE , J., K ARIMI , A., B ORDES , A., AND W ESTON , J. 2016. Key-value memory networks for directly reading documents. In 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), J. Su, X. Carreras, and K. Duh, Eds. The Association for Computational Linguistics, Austin, TX, 1400–1409. find
  9. N AVIGLI , R. AND P ONZETTO , S. P. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence 193, 217–250. find
  10. O BJECT -M ANAGEMENT -G ROUP . 2017. Semantics of business vocabulary and business rules (SBVR), v. 1.4. OMG standards document. http://www.omg.org/spec/SBVR/Current. ↗
  11. R INGGAARD , M., G UPTA , R., AND P EREIRA , F. C. N. 2017. SLING: A framework for frame semantic parsing. CoRR 1710.07032, 1–9. find
  12. S AHA , D., F LORATOU , A., S ANKARANARAYANAN , K., M INHAS , U. F., M ITTAL , A. R., AND Ö ZCAN , F. 2016. ATHENA: an ontology-driven system for natural language querying over relational data stores. PVLDB 9, 12, 1209–1220. find
  13. S CHWITTER , R. 2010. Controlled natural languages for knowledge representation. In COLING 2010, 23rd Intl. Conf. on Computational Linguistics, Posters Volume, 23-27. ACL, Beijing, China, 1113–1121. find
  14. Z HANG , Y., D AI , H., K OZAREVA , Z., S MOLA , A. J., AND S ONG , L. 2018a. The MetaQA dataset. https://github.com/yuyuz/MetaQA. ↗
  15. Z HANG , Y., D AI , H., K OZAREVA , Z., S MOLA , A. J., AND S ONG , L. 2018b. Variational reasoning for question answering with knowledge graph. See McIlraith and Weinberger (2018), 6069–6076. Querying Knowledge via Multi-Hop English Questions

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gao-2019-querying-knowledge-multi-hop-english/