Elamur Research⌕
☰
home / papers / 2022 / Knowledge Authoring with Factual English

Knowledge Authoring with Factual English

Paper · 2022 · Yuheng Wang, Giorgian Borca-Tasciuc, Nikhil Goel et al.

Summary

Knowledge representation and reasoning systems need facts written in logic, which domain experts rarely can produce. Free-text extractors are too error-prone for reasoning, and controlled natural languages such as Attempto's ACE are tedious to learn. KALM, an earlier ACE-based system, avoided "semantic mismatch": it gave paraphrases the same logical form by using FrameNet-style frames and BabelNet senses. This paper moves KALM onto a neural dependency parser, mStanza, a modified Stanza that returns several ranked parses. The authors also add fixes for the tagging, lemmatization and dependency errors that neural parsers make. The resulting system, KALM_FL, handles "factual" English, meaning statements and queries that convey facts. Commands and other non-factual material are excluded or ignored. KALM_FL was tested on four benchmarks: CNLD, a tense/voice-perturbed CNLDM, MetaQA and NLD. It scores about 0.95 or higher on all of them, nearly matching the original KALM on CNL input while also covering text that KALM cannot parse. It clearly beats OpenSesame and also outperforms DrKIT on MetaQA. The work shows that frame-based authoring can drop most CNL restrictions without losing much accuracy.

Key points

  • The pipeline goes from parse to DRS-like facts, then to logical valence patterns learned from annotated training sentences, then to frame-based parsing. Role fillers are disambiguated through BabelNet, and the output is a unique logical representation built from frame/role/synset/text facts.
  • KALM_FL frame/role/synset Micro-F1 scores:
    • CNLD: 0.99/0.99/0.97
    • CNLDM: 0.99/0.99/0.97
    • MetaQA: 0.95/0.95/0.95
    • NLD: 0.99/0.98/0.95
  • OpenSesame performs far worse. Its frame F1 is 0.49–0.61, and its role F1 is 0.00–0.17.
  • On MetaQA, 333 of 350 templates were parsed correctly. These cover 128,784 2-hop and 119,923 3-hop queries. KALM_FL reaches 0.962 and 0.933 accuracy on them, against DrKIT's 0.871 and 0.876.
  • Current limitations:
    • Temporal information is accepted but discarded.
    • There is no anaphora resolution.
    • Quantified sentences such as "every pet has an owner" are treated as facts rather than rules.

Interesting ideas

  • I1. Each frame-role is learned as a short syntactic path from the trigger word, such as verb→subject for Buyer, taken from a single annotated example and reused for synonymous triggers. This makes role extraction transparent and easy to inspect.
  • I2. Word senses are chosen by a weighted breadth-first search through BabelNet, starting at the role's own synset and keeping the cheapest-reached sense of the filler. This makes the role constrain the sense.
  • I3. The parser emits several ranked analyses instead of one, so later semantic stages can recover when the top parse is wrong.
  • I4. The meaning-preserving perturbation set (CNLDM, with tense and voice changed) tests whether the logical output stays invariant across paraphrases.
  • I5. The metrics are strict and sentence-level: a sentence counts only if all frames, then all roles, then all senses are correct, with no spurious frames.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Store VerbNet role mappings as explicit UD-path patterns per verb class, so every role assignment can be traced to the path that produced it.
  • I2 (world): When linking entities to WordNet/Wikidata nodes, rank candidate senses by graph distance from the frame role's type, which gives an explainable choice.
  • I3 (tree): Let gparse or the perceptron parser pass k-best trees to frame matching, and pick the best tree by how well it fits the frames.
  • I4 (learn): Build invariance controls by rewriting sentences into passive or other tenses, then check that the frames and world state stay identical.
  • I5 (answer): Report tiered all-or-nothing scores per sentence (frame, role, sense) alongside token-level ones, to expose where meaning breaks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stony Brook UniversityStony Brook · United States27 works · 2,057 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 English · 1,373 works

Cite

Cite as Yuheng Wang, Giorgian Borca-Tasciuc, Nikhil Goel, Paul Fodor and Michael Kifer. 2022. Knowledge Authoring with Factual English. arXiv:2208.03094 · EPTCS 364, 2022, pp. 107-122.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.4204/eptcs.364.10 · arXiv 2208.03094

@misc{wang2022knowledge,
  title = {{Knowledge Authoring with Factual English}},
  author = {Yuheng Wang, Giorgian Borca-Tasciuc, Nikhil Goel, Paul Fodor and Michael Kifer},
  year = {2022},
  doi = {10.4204/eptcs.364.10},
  eprint = {2208.03094},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.4204/eptcs.364.10}
}

References

14 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Collin F Baker, Charles J Fillmore & John B Lowe (1998): The berkeley framenet project. In: 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1, pp. 86–90, doi:10.3115/980845.980860. ↗ our card
  2. Dipanjan Das, Desai Chen, André FT Martins, Nathan Schneider & Noah A Smith (2014): Frame-semantic parsing. Computational linguistics 40(1), pp. 9–56, doi:10.1162/COLI_a_00163. ↗ our card
  3. Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Ruslan Salakhutdinov & William W Cohen (2020): Differentiable reasoning over a virtual knowledge base. arXiv preprint arXiv:2002.10640 1, doi:10.48550/arXiv.2002.10640. ↗
  4. Charles J Fillmore et al. (2006): Frame semantics. Cognitive linguistics: Basic readings 34, pp. 373–400, doi:10.1515/9783110199901.373. ↗
  5. Norbert E Fuchs, Stefan Hoefler, Kaarel Kaljurand, Tobias Kuhn, Gerold Schneider & Uta Schwertel (2006): Discourse Representation Structures for ACE 5. ifi Technical Reports 1(ifi2006. 10), doi:10.5167/ uzh-62058. find
  6. Norbert E Fuchs & Rolf Schwitter (1996): Attempto controlled english (ace). arXiv preprint cmp-lg/9603003 1, doi:10.48550/arXiv.cmp-lg/9603003. ↗
  7. Tiantian Gao, Paul Fodor & Michael Kifer (2018): High accuracy question answering via hybrid controlled natural language. In: 2018 IEEE/WIC/ACM International Conference on Web Intelligence (WI), IEEE, pp. 17–24, doi:10.1109/WI.2018.0-112. ↗
  8. Tiantian Gao, Paul Fodor & Michael Kifer (2018): Knowledge authoring for rule-based reasoning. In: OTM Confederated International Conferences” On the Move to Meaningful Internet Systems”, Springer, pp. 461–480, doi:10.1007/978-3-030-02671-4_28. ↗
  9. Tiantian Gao, Paul Fodor & Michael Kifer (2019): Querying Knowledge via Multi-Hop English Questions. Theory and Practice of Logic Programming 19(5-6), pp. 636–653, doi:10.1017/S1471068419000103. ↗ our card
  10. Roberto Navigli & Simone Paolo Ponzetto (2012): BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence 193, pp. 217–250, doi:10.1016/j.artint.2012.07.001. ↗
  11. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton & Christopher D Manning (2020): Stanza: A python natural language processing toolkit for many human languages. arXiv preprint arXiv:2003.07082 1, doi:10. 18653/v1/2020.acl-demos.14. ↗ our card
  12. Michael Ringgaard, Rahul Gupta & Fernando CN Pereira (2017): SLING: A framework for frame semantic parsing. arXiv preprint arXiv:1710.07032 1, doi:10.48550/arXiv.1710.07032. ↗
  13. Swabha Swayamdipta, Sam Thomson, Chris Dyer & Noah A Smith (2017): Frame-semantic parsing with softmax-margin segmental rnns and a syntactic scaffold. arXiv preprint arXiv:1706.09528 1, doi:10.48550/ arXiv.1706.09528. ↗ our card
  14. Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J Smola & Le Song (2018): Variational reasoning for question answering with knowledge graph. In: Thirty-second AAAI conference on artificial intelligence, pp. 6069–6076, doi:10.48550/arXiv.1709.04071. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2022-knowledge-authoring-factual-english/