Elamur Research⌕
☰
home / papers / 2022 / Universal Proposition Bank 2.0

Universal Proposition Bank 2.0

Paper · 2022 · Ishan Jindal, Alexandre Rademaker, Michał Ulewicz et al.

Summary

Most languages have no propbank because manual annotation is expensive. The non-English propbanks that do exist use their own rolesets, which makes a single multilingual semantic role labeler impractical. UP2.0 extends the authors' 2017 resource by automatically labeling UD treebank text in 23 languages (8 families, UD 2.9) with English PropBank frames and roles. English SRL output is carried over to target sentences through parallel corpora in two stages. First, a filtered projection favors precision. Then repeated bootstrapped retraining of a target-language labeler recovers recall.

The main upgrades over UP1.0 are:

  • a new English neural labeler (NNSRL) that predicts argument heads and spans together;
  • Stanza for parsing and SimAlign for word alignment;
  • a hybrid step that adds predicates found by an older syntax-based labeler.

Arguments now carry spans as well as heads, so syntax-agnostic models can be trained. Small hand-checked test sets exist for Polish, Portuguese and English. On these gold sets the authors report clear quality gains over UP1.0, which makes the release a common benchmark for cross-lingual SRL.

Key points

  • Coverage grew from 7 treebanks (UD 1.4) in UP1.0 to 23 languages on UD 2.9, with both span- and dependency-based roles. Gold annotations exist for Polish, Portuguese and English, published under github.com/UniversalPropositions.
  • The release is a separate per-sentence semantic layer (predicates, senses, head and span arguments) plus a Python script that merges it with UD files, so the SRL data does not depend on UD revisions.
  • Parser choice was based on UD 2.5 test sets for 8 languages. Stanza won in most of them; for English its LAS was 87.03, against 80.62 for UDPipe and 44.71 for spaCy trf. For ES and ZH Stanza was close to the best.
  • The aligners compared were BerkeleyAligner, Awesome-align and SimAlign. Both neural aligners clearly beat the statistical one. SimAlign/itermax was chosen because aligner recall mattered most for projection quality (its EN-FR recall was 95.10).
  • Bitext came from EP/OS/CC combined with TB, ranging from 0.38M to 3.69M sentence pairs per language before filtering. Two filters were applied:
    • sentences that Stanza split into several were dropped;
    • multi-word tokens were removed to avoid alignment problems.

Interesting ideas

  • I1. NNSRL has two branches, one for argument heads and one for spans, trained with a summed loss. At inference an argument survives only if its predicted head falls inside its predicted span, giving a cheap consistency filter.
  • I2. The authors explain why spans cannot be derived from heads by rule: some relations stay inside the entity (amod) and some do not (nsubj of a nominal predicate); others are ambiguous (dep) or treebank-specific; and in coordination one token can head two different entities.
  • I3. Projection is built precision-first by accepting only well-filtered labels; recall is recovered afterwards through repeated retraining, which keeps noise from compounding.
  • I4. A precise neural labeler that misses predicates is combined with a syntax-based labeler that finds nearly all of them. Only the high-recall component (predicate detection) is borrowed from the weaker system.
  • I5. Annotations are shipped as a detached layer with a merge script, so the resource survives tokenization and guideline changes in the underlying treebank.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): prag could predict a role's span and check that the UD head chosen for the role lies inside it, flagging mismatches as explainable errors instead of silently accepting them.
  • I2 (world): entity extraction from the tree should treat coordination and dep as explicit special cases rather than assuming "subtree = entity".
  • I3 (learn): silver data could first keep only sentences where all witness parsers agree, then retrain and widen coverage step by step while watching precision on negative controls.
  • I4 (frames): predicate candidates from rule-based UD patterns could supplement a learned frame detector, with the source of each predicate recorded for explanation.
  • I5 (tree): frame or role annotations on treebanks could be stored apart from the UD files and re-attached after dialect conversion or treebank updates.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IBMArmonk · United States85 works · 66,118 citations🏢AppleCupertino · United States10 works · 1,972 citations

Also on the byline: Vietnam National University, Hanoi.

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 3 English · 1,373 works Semantic role labeling · 132 works Universal Dependencies · 172 works

Cite

Cite as Ishan Jindal, Alexandre Rademaker, Michał Ulewicz, Ha Linh, Huyen Nguyen, Khoi-Nguyen Tran, Huaiyu Zhu and Yunyao Li. 2022. Universal Proposition Bank 2.0. Proceedings of the Thirteenth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{jindal2022universal,
  title = {{Universal Proposition Bank 2.0}},
  author = {Ishan Jindal, Alexandre Rademaker, Michał Ulewicz, Ha Linh, Huyen Nguyen, Khoi-Nguyen Tran, Huaiyu Zhu and Yunyao Li},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/4e5ta5tbonkz},
  url = {https://doi.org/10.63317/4e5ta5tbonkz}
}

References

50 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Afonso, S., Bick, E., Haber, R., and Santos, D. (2002). Floresta sintá (c) tica: a treebank for portuguese. In Proceedings of the Third International Conference on Language Resources and Evaluation (LREC 2002). ELRA. 11 https://ufal.mff.cuni.cz/deep-universal-dependencies ↗
  2. Ak, K. and Yildiz, O. T. (2019). Automatic propbank generation for turkish. In RANLP.
  3. Akbik, A. and Li, Y. (2016). K-srl: instance-based learning for semantic role labeling. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 599–608. find
  4. Akbik, A., Chiticariu, L., Danilevsky, M., Li, Y., Vaithyanathan, S., and Zhu, H. (2015). Generating high quality proposition banks for multilingual semantic role labeling. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 397–407.
  5. Cai, R. and Lapata, M. (2020). Alignment-free cross-lingual semantic role labeling. In EMNLP. find
  6. Conia, S., Bacciu, A., and Navigli, R. (2021). Unifying cross-lingual semantic role labeling with heterogeneous linguistic resources. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 338–351, Online, June. Association for Computational Linguistics. find
  7. Dou, Z.-Y. and Neubig, G. (2021). Word alignment by fine-tuning embeddings on parallel corpora. ↗
  8. Droganova, K. and Zeman, D. (2019). Towards deep universal dependencies. In Proceedings of the Fifth International Conference on Dependency Linguistics (Depling, SyntaxFest 2019), pages 144–152. find
  9. Exner, P., Klang, M., and Nugues, P. (2016). Multilingual supervision of semantic annotation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1007–1017. find
  10. Fei, H., Zhang, M., Li, F., and Ji, D. (2020). Cross-lingual semantic role labeling with model transfer. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2427–2437. ↗
  11. Gotham, M. and Haug, D. T. T. (2019). Glue semantics for universal dependencies. In Proceedings of the LFG’18 Conference. CSLI Publications. find
  12. Guan, C., Cheng, Y., and Zhao, H. (2019). Semantic role labeling with associated memory network. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3361– 3371, Minneapolis, Minnesota, June. Association for Computational Linguistics. find
  13. Gunasekara, S., Chathura, D., Jeewantha, C., and Dias, G. (2020). Using annotation projection for semantic role labeling of low-resourced language: Sinhala. 2020 International Conference on Asian Language Processing (IALP), pages 98–103. find
  14. Jindal, I., Aharonov, R., Brahma, S., Zhu, H., and Li, Y. (2020a). Improved semantic role labeling using parameterized neighborhood memory adaptation. arXiv preprint arXiv:2011.14459. ↗
  15. Jindal, I., Li, Y., Brahma, S., and Zhu, H. (2020b). Clar: A cross-lingual argument regularizer for semantic role labeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 3113–3125. find
  16. Jurafsky, D. and Martin, J. H. (2021). Speech and language processing. Draft of December 29. find
  17. Maqsud, U., Arnold, S., Hülfenhaus, M., and Akbik, A. (2014). Nerdle: Topic-specific question answering using wikia seeds. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: System Demonstrations, pages 81–85. find
  18. Mille, S., Belz, A., Bohnet, B., and Wanner, L. (2018). Underspecified Universal Dependency structures as inputs for multilingual surface realisation. In Proceedings of the 11th International Conference on Natural Language Generation, pages 199–209, Tilburg University, The Netherlands, November. Association for Computational Linguistics.
  19. Mulcaire, P., Kasai, J., and Smith, N. A. (2019). Polyglot contextual representations improve crosslingual transfer. arXiv preprint arXiv:1902.09697. ↗
  20. Niklaus, C., Cetto, M., Freitas, A., and Handschuh, S. (2018). A survey on open information extraction. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3866– 3878, Santa Fe, New Mexico, USA, August. Association for Computational Linguistics. find
  21. Och, F. J. and Ney, H. (2003). A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. ↗
  22. Ouchi, H., Shindo, H., and Matsumoto, Y. (2018). A span selection model for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1630–1642, Brussels, Belgium, Oct. Association for Computational Linguistics. find
  23. Padó, S. and Lapata, M. (2009). Cross-lingual annotation projection for semantic roles. Journal of Artificial Intelligence Research, 36:307–340. ↗
  24. Padó, S. (2007). Cross-lingual annotation projection models for role-semantic information. Saarland University. find
  25. Qi, P., Zhang, Y., Zhang, Y., Bolton, J., and Manning, C. D. (2020). Stanza: A python natural language processing toolkit for many human languages. arXiv preprint arXiv:2003.07082. ↗
  26. Sabet, M. J., Dufter, P., Yvon, F., and Schütze, H. (2020). Simalign: High quality word alignments without parallel training data using static and contextualized embeddings. arXiv preprint arXiv:2004.08728. ↗
  27. Sabet, M. J., Dufter, P., Yvon, F., and Schütze, H. (2021). Simalign: High quality word alignments without parallel training data using static and contextualized embeddings. ↗
  28. Schuster, S. and Manning, C. D. (2016). Enhanced English Universal Dependencies: An improved representation for natural language understanding tasks. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2371–2378, Portorož, Slovenia, May. European Language Resources Association (ELRA).
  29. Shi, P. and Lin, J. (2019). Simple bert models for relation extraction and semantic role labeling. arXiv preprint arXiv:1904.05255. ↗
  30. Shi, C., Liu, S., Ren, S., Feng, S., Li, M., Zhou, M., Sun, X., and Wang, H. (2016). Knowledge-based semantic embedding for machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2245–2254. ↗
  31. Straka, M. (2018). UDPipe 2.0 prototype at CoNLL 2018 UD shared task. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 197–207, Brussels, Belgium, October. Association for Computational Linguistics.
  32. Van der Plas, L., Merlo, P., and Henderson, J. (2011). Scaling up automatic cross-lingual semantic role annotation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 299– 304. find
  33. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems, pages 5998– 6008.
  34. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019). Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, pages arXiv–1910. ↗
  35. Yih, W.-t., Richardson, M., Meek, C., Chang, M.-W., and Suh, J. (2016). The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201–206.
  36. Zhang, H., Wang, H., and Roth, D. (2020a). Unsupervised label-aware event trigger and argument classification. CoRR, abs/2012.15243. ↗
  37. Zhang, Z., Wu, Y., Zhao, H., Li, Z., Zhang, S., Zhou, X., and Zhou, X. (2020b). Semantics-aware bert for language understanding. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9628– 9635. 8. find
  38. Language Resource References Duran, M. S. and Aluı́sio, S. (2011). Propbank-br: a brazilian portuguese corpus annotated with semantic role labels. In Proceedings of the 8th Brazilian Symposium in Information and Human Language Technology. find
  39. Erk, K., Kowalski, A., Padó, S., and Pinkal, M. (2003). Towards a resource for lexical semantics: A large german corpus with extensive semantic annotation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 537– 544. find
  40. Fillmore, C. J., Ruppenhofer, J., and Baker, C. F. (2004). Framenet and representing the link between semantic and syntactic relations. Frontiers in linguistics, 1:19–59. find
  41. Haverinen, K., Kanerva, J., Kohonen, S., Missilä, A., Ojala, S., Viljanen, T., Laippala, V., and Ginter, F. (2015). The finnish proposition bank. Language Resources and Evaluation, 49(4):907–926. ↗
  42. Koehn, P. et al. (2005). Europarl: A parallel corpus for statistical machine translation. In MT summit, volume 5, pages 79–86. Citeseer. find
  43. Lison, P. and Tiedemann, J. (2016). Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles.
  44. Meyers, A., Reeves, R., Macleod, C., Szekely, R., Zielinska, V., Young, B., and Grishman, R. (2004). The nombank project: An interim report. In Proceedings of the workshop frontiers in corpus annotation at hlt-naacl 2004, pages 24–31.
  45. Palmer, M., Gildea, D., and Kingsbury, P. (2005). The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–106. ↗
  46. Rademaker, A., Chalub, F., Real, L., Freitas, C., Bick, E., and de Paiva, V. (2017). Universal dependencies for portuguese. In Proceedings of the Fourth International Conference on Dependency Linguistics (Depling), pages 197–206, Pisa, Italy, September. find
  47. Tiedemann, J. (2012). Parallel data, tools and interfaces in opus. In Lrec, volume 2012, pages 2214– 2218. Citeseer. find
  48. Vaidya, A., Choi, J. D., Palmer, M., and Narasimhan, B. (2012). Empty argument insertion in the hindi propbank. In LREC, pages 1522–1526. find
  49. Weischedel, R., Palmer, M., Marcus, M., Hovy, E., Pradhan, S., Ramshaw, L., Xue, N., Taylor, A., Kaufman, J., Franchini, M., et al. (2013). Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23. find
  50. Zaghouani, W., Diab, M., Mansouri, A., Pradhan, S., and Palmer, M. (2010). The revised arabic propbank. In Proceedings of the fourth linguistic annotation workshop, pages 222–226. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jindal-2022-universal-proposition-bank-2-0/