Comparing Sense Categorization Between English PropBank and English WordNet

What it is about

The authors hand-tag verb senses in Penn Treebank sentences with WordNet senses, and compare the result with the senses already given in English PropBank. The sample is 7,576 sentences containing 1,554 senses of 1,453 verbs. For most PropBank senses (1,184 senses of 1,118 verbs) there is a one-to-one match in WordNet. In 352 senses of 329 verbs, one PropBank sense splits into up to 12 WordNet senses, often because of a syntactic difference such as transitive and intransitive use. In a smaller number of cases, several PropBank senses merge into one WordNet sense (6 verbs) or overlap (29 verbs). The authors also compare their manual tags with an automatically built resource: they found 418 matching senses, but 1,721 senses with differences, and they conclude that manual tagging still catches systematic errors that automatic alignment misses.

Ideas

I1. world/src/dictionary.rs: make a cross-table between Mova's dictionary senses and WordNet synsets for the verbs in the school lessons. Count one-to-one, one-to-many and many-to-one matches as the paper does. The cells with many-to-one matches are the places where the lexicon needs a new sense.
I2. Add the collocations the paper annotates (for example get up, buy back) as multi-word entries in en/src/dict.rs, and check whether the sense-level accuracy on the school tests rises. The paper finds collocation senses for 18 senses of 15 verbs.
I3. Use two independent passes by two authors on a sample of verbs, and report agreement before any discussion. Disagreements become the list of senses to revise, not errors to fix silently.

For Mova

The chapter is languages/en/19-lexicon-mwe-register.md, with languages/en/13-verbs.md for verb senses. The code is world/src/dictionary.rs (word senses), en/src/dict.rs and en/src/tok.rs (multi-word matching). The measures: the one-to-one share on the lesson verbs, the sense accuracy on the school tests, and the agreement between two passes. The paper's data are not Mova's; its method of cross-checking two sense inventories is the part to use.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written by Mova (claude-opus-5-5) from the paper text. Original — aclanthology.org/2019.gwc-1.39.pdf

Summary

The authors examine how closely the division of verbs into senses in English PropBank matches the sense system of Princeton WordNet. To do this, they took 7576 sentences from the Penn Treebank. These sentences contain 1453 distinct verbs with 1554 PropBank senses. Each verb was manually annotated with PWN 3.1 synsets in a custom Java tool. The tool displays the syntactic tree of the sentence and suggests candidate senses. It also reduces word forms to their stems and takes fixed expressions such as "get up" into account. Two linguist annotators saw all examples of a given verb together. They first proofread the annotation and then determined the mapping type: one-to-one, one-to-many, many-to-one, or many-to-many. The vast majority of PropBank senses have a direct counterpart in WordNet, but many of them are split or merged. Separately, the manual annotation was compared with the automatically constructed Predicate Matrix. The comparison revealed missing verbs, missing senses, and erroneous mergers. The authors conclude that manual annotation, at least in combination with automatic annotation, gives a higher-quality result. They plan to use the resource to extend the Turkish PropBank.

Key points

Data: 7576 sentences from the Penn Treebank, 1554 PropBank senses for 1453 verbs. The sense inventory is PWN 3.1, converted into a BalkaNet-style XML format. The annotation tool DataCollector is openly available on GitHub.
For 1184 senses (1118 verbs), an exact one-to-one mapping was found. Mismatches have three causes: WordNet divides senses more finely, merges several senses into one, or one of its senses covers several PropBank senses.
Comparison with Predicate Matrix: discrepancies occur in 1721 senses (1387 verbs). For 307 senses (281 verbs), none of the assigned senses match, and for 199 senses (178 verbs), matches are combined with discrepancies.
Predicate Matrix contains 593 senses that are absent from the manual annotation and additional senses for 497 entries. Conversely, the manual annotation contains 125 senses (123 verbs) missing from PM. Eight verbs, among them emote and encrypt, are not annotated in PM at all. PM also erroneously merges distinct senses, such as appeal 01 and appeal 03.
The authors plan to extend the work to the Turkish Proposition Bank (Ak et al., 2018), where 9560 translated sentences and 1914 verb senses have already been annotated.

Topics: Semantic role labeling

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2019.gwc-1.39

@inproceedings{bakay2019comparing,
  title = {{Comparing Sense Categorization Between English PropBank and English WordNet}},
  author = {Bakay, Özge and Avar, Begüm and Yıldız, Olcay Taner},
  booktitle = {Proceedings of the 10th Global Wordnet Conference},
  year = {2019},
  doi = {10.18653/v1/2019.gwc-1.39},
  url = {https://doi.org/10.18653/v1/2019.gwc-1.39}
}

References

16 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

K. Ak, C. Toprak, V. Esgel, and O. T. Yildiz. 2018. Construction of Turkish proposition bank. Turkish Journal of Electrical Engineering & Computer Sciences. find
O. Babko-Malaya, 2005. Framers. Guidelines for Propbank C. F. Baker, C. J. Fillmore, and J. B. Lowe. 1998. The Berkeley FrameNet project. In COLING-ACL ’98: Proceedings of the Conference, pages 86–90, Montreal, Canada. find our card
M. Lopez de Lacalle, E. Laparra, and G. Rigau. 2014a. First steps towards a predicate matrix. In Proceedings of the 7th Global WordNet Conference, Tartu, Estonia, Jan 25-29. find
M. Lopez de Lacalle, E. Laparra, and G. Rigau. 2014b. Predicate matrix: extending semlink through wordnet mappings. In Proceedings of the Eighth International Conference on Language Resources and Evaluation. find
M. Lopez de Lacalle, E. Laparra, I. Aldabe, and G. Rigau. 2016a. A multilingual predicate matrix. In Proceedings of the Tenth International Conference on Language Resources and Evaluation, Paris, France, may. find
M. Lopez de Lacalle, E. Laparra, I. Aldabe, and G. Rigau. 2016b. Predicate matrix: automatically extending the semantic interoperability between predicate resources. Language Resources and Evaluation, 50(2):263–289, Jun. find
C. Fellbaum. 1998. ed. wordnet: an electronic lexical database. MIT Press, Cambridge MA, 1:998. find
P. Kingsbury and M. Palmer. 2002. From treebank to propbank. In LREC. European Language Resources Association. find our card
P. Kingsbury and M. Palmer. 2003. Propbank: The next level of treebank. In Proceedings of Treebanks and Lexical Theories, Växjö, Sweden. find
K. Kipper-Schuler. 2005. VerbNet: a broad-coverage, comprehensive verb lexicon. Ph.D. thesis, Computer and Information Science Department, University of Pennsylvania, Philadelphia, PA. find
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41. ↗ our card
M. Palmer, D. Gildea, and P. Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Comput. Linguist., 31(1):71–106, March. find our card
M. Palmer. 2009. Semlink: Linking propbank, verbnet and framenet. In Proceedings of the Generative Lexicon Conference, Pisa, Italy. find
M. T. Pazienza, M. Pennacchiotti, and F. M. Zanzotto. 2006. Mixing wordnet, verbnet and propbank for studying verb relations. 01. find
J. I. Saeed. 1997. Semantics. Blackwell. find
S. Stamou, K. Oflazer, K. Pala, D. Christodoulakis, D. Cristea, D. Tufis, S. Koeva, S. Totkov, D. Dutoit, and M. Grigoriadou. 2002. Balkanet: A multilingual semantic network for balkan languages. In Proceedings of the First International WordNet Conference, pages 21–25, Mysore, India. find