Elamur Research
☰
home / papers / 2015 / Generating High Quality Proposition Banks for Multilingual Semantic Role Labeling

Generating High Quality Proposition Banks for Multilingual Semantic Role Labeling

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P15-1039.pdf

Summary

Semantic role labeling needs PropBank-style training data, but such annotations are costly and missing for most languages. Earlier work transferred English labels to other languages through word-aligned parallel corpora. This direct projection breaks down when translations are not literal, for example when a verb is rendered as a noun or with a different meaning. Akbik and colleagues analyzed these projection errors on a French gold standard and built a two-stage pipeline. First, filters keep only reliable projected labels. A verb filter, a translation-dictionary filter and a reattachment heuristic together produce a target corpus with high precision but low recall. Second, a bootstrap procedure repeatedly retrains an SRL model on fully labeled target sentences and adds new labels, which raises recall while largely preserving precision. The method needs no language-specific tuning. It outperformed direct projection and was used to build PropBanks for 7 languages from three language families, which the authors share with researchers on request. This makes it practical to train SRL systems for resource-poor languages.

Key points

  • Evaluation data: the French gold set contains 1,000 English–French Europarl sentence pairs with manual PropBank labels. English labels were produced with CLEARNLP.
  • Recall ceiling: only about 45% of French predicate labels and 35% of argument labels have an English counterpart, which caps the recall of any projection method.
  • Main error types: translation shifts account for 37% (predicate mismatch) and 36% (verb→non-verb) of predicate errors. Labels attached to a non-head word cause 33% of argument errors.
  • Effect of the filters: direct projection reaches F1 of 0.43 for predicates and 0.36 for arguments. TF+RH lifts precision to 0.88 for predicates and 0.75 for arguments, and VF+RH reaches an argument F1 of 0.46.
  • Additional experiments:
    • Shrinking the French corpus to 250k sentences changed results only slightly.
    • Using a stronger English SRL system (MATE-SRL vs. CLEARNLP) noticeably affected argument quality.
    • A simple heuristic for French expletive constructions gave a small precision gain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semantic role labeling

Cited by

63 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 59 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p15-1039

@inproceedings{akbik2015generating,
  title = {{Generating High Quality Proposition Banks for Multilingual Semantic Role Labeling}},
  author = {Akbik, Alan and Chiticariu, Laura and Danilevsky, Marina and Li, Yunyao and Vaithyanathan, Shivakumar and Zhu, Huaiyu},
  booktitle = {Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2015},
  doi = {10.3115/v1/p15-1039},
  url = {https://doi.org/10.3115/v1/p15-1039}
}

References

36 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The berkeley framenet project. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics-Volume 1, pages 86–90. Association for Computational Linguistics. find our card
  2. Roberto Basili, Diego De Cao, Danilo Croce, Bonaventura Coppola, and Alessandro Moschitti. 2009. Cross-language frame semantics transfer in bilingual corpora. In Computational Linguistics and Intelligent Text Processing, pages 332–345. Springer. find
  3. Anders Björkelund, Love Hafdell, and Pierre Nugues. 2009. Multilingual semantic role labeling. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning: Shared Task, pages 43–48. Association for Computational Linguistics. find
  4. Bernd Bohnet and Joakim Nivre. 2012. A transition-based system for joint part-of-speech tagging and labeled non-projective dependency parsing. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1455–1465. Association for Computational Linguistics. find
  5. Bernd Bohnet. 2010. Very high accuracy and fast dependency parsing is not a contradiction. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 89–97. Association for Computational Linguistics. find our card
  6. Ondřej Bojar, Vojtěch Diatka, Pavel Rychlỳ, Pavel Straňák, Vı́t Suchomel, Aleš Tamchyna, Daniel Zeman, et al. 2014. Hindencorp–hindi-english and hindi-only corpus for machine translation. In Proceedings of the Ninth International Conference on Language Resources and Evaluation. find
  7. Jinho D. Choi and Andrew McCallum. 2013. Transition-based dependency parsing with selectional branching. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. find
  8. Laurence Danlos. 2005. Automatic recognition of french expletive pronoun occurrences. In Natural language processing. Proceedings of the 2nd International Joint Conference on Natural Language Processing (IJCNLP-05), pages 73–78. Citeseer. find
  9. John DeNero and Percy Liang. 2007. The Berkeley Aligner. http://code.google.com/p/berkeleyaligner/. ↗
  10. K. Erk, A. Kowalski, S. Pado, and S. Pinkal. 2003. Towards a resource for lexical semantics: A large german corpus with extensive semantic annotation. In ACL. find
  11. Spence Green and Christopher D Manning. 2010. Better arabic parsing: Baselines, evaluations, and analysis. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 394–402. Association for Computational Linguistics. find our card
  12. Jan Hajič, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Antònia Martı́, Lluı́s Màrquez, Adam Meyers, Joakim Nivre, Sebastian Padó, Jan Štěpánek, et al. 2009. The conll-2009 shared task: Syntactic and semantic dependencies in multiple languages. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–18. Association for Computational Linguistics. find our card
  13. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT summit, volume 5, pages 79–86. find
  14. Mikhail Kozhevnikov and Ivan Titov. 2013. Cross-lingual transfer of semantic role labeling models. In ACL (1), pages 1190–1200. find
  15. Umar Maqsud, Sebastian Arnold, Michael Hülfenhaus, and Alan Akbik. 2014. Nerdle: Topic-specific question answering using wikia seeds. In Lamia Tounsi and Rafal Rak, editors, COL-ING 2014, 25th International Conference on Computational Linguistics, Proceedings of the Conference System Demonstrations, August 23-29, 2014, Dublin, Ireland, pages 81–85. ACL. find
  16. Paola Merlo and Lonneke van der Plas. 2009. Abstraction and generalisation in semantic role labels: Propbank, verbnet or both? In ACL 2009, pages 288–296. find our card
  17. Adam Meyers, Ruth Reeves, Catherine Macleod, Rachel Szekely, Veronika Zielinska, Brian Young, and Ralph Grishman. 2004. Annotating noun argument structure for nombank. In LREC, volume 4, pages 803–806. find
  18. Paola Monachesi, Gerwert Stevens, and Jantine Trapman. 2007. Adding semantic role annotation to a corpus of written dutch. In Proceedings of the Linguistic Annotation Workshop, LAW ’07, pages 77–84. find
  19. Joakim Nivre, Johan Hall, and Jens Nilsson. 2006. Maltparser: A data-driven parser-generator for dependency parsing. In Proceedings of LREC, volume 6, pages 2216–2219. find
  20. Sebastian Padó and Mirella Lapata. 2009. Cross-lingual annotation projection for semantic roles. Journal of Artificial Intelligence Research, 36(1):307–340. find
  21. Sebastian Pado. 2007. Cross-Lingual Annotation Projection Models for Role-Semantic Information. Ph.D. thesis, Saarland University. MP. find
  22. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational linguistics, 31(1):71–106. ↗ our card
  23. Alexandre Rafalovitch, Robert Dale, et al. 2009. United nations general assembly resolutions: A six-language parallel corpus. In Proceedings of the MT Summit, volume 12, pages 292– 299. find
  24. Karin Kipper Schuler. 2005. Verbnet: A Broad-coverage, Comprehensive Verb Lexicon. Ph.D. thesis, University of Pennsylvania. find
  25. Dan Shen and Mirella Lapata. 2007. Using semantic roles to improve question answering. In EMNLP-CoNLL, pages 12–21. Citeseer. find
  26. Oscar Täckström, Dipanjan Das, Slav Petrov, Ryan McDonald, and Joakim Nivre. 2013. Token and type constraints for cross-lingual part-of-speech tagging. Transactions of the Association for Computational Linguistics, 1:1–12. find
  27. Ivan Titov and Alexandre Klementiev. 2012. Crosslingual induction of semantic roles. In ACL, pages 647–656. find
  28. Ashwini Vaidya, Jinho D Choi, Martha Palmer, and Bhuvana Narasimhan. 2011. Analysis of the hindi proposition bank using dependency structure. In Proceedings of the 5th Linguistic Annotation Workshop, pages 21–29. Association for Computational Linguistics. find
  29. Lonneke Van der Plas, Tanja Samardžić, and Paola Merlo. 2010. Cross-lingual validity of propbank in the manual annotation of french. In Proceedings of the Fourth Linguistic Annotation Workshop, pages 113–117. Association for Computational Linguistics. find
  30. Lonneke Van der Plas, Paola Merlo, and James Henderson. 2011. Scaling up automatic cross-lingual semantic role annotation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 299–304. Association for Computational Linguistics. find
  31. Lonneke Van der Plas, Marianna Apidianaki, Rue John von Neumann, and Chenhua Chen. 2014. Global methods for cross-lingual semantic role and predicate labelling. In Proceedings of the 25th International Conference on Computational Linguistics (COLING 2014), pages 1279–1290. Association for Computational Linguistics. find
  32. George Weber. 1997. Top languages: The world’s 10 most influential languages. Language Today, December. find
  33. Nianwen Xue. 2006. Semantic role labeling of nominalized predicates in chinese. In Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, pages 431– 438. Association for Computational Linguistics. find
  34. Nianwen Xue. 2008. Labeling chinese predicates with semantic roles. Computational linguistics, 34(2):225–255. find
  35. David Yarowsky, Grace Ngai, and Richard Wicentowski. 2001. Inducing multilingual text analysis tools via robust projection across aligned corpora. In Proceedings of the first international conference on Human language technology research, pages 1–8. Association for Computational Linguistics. find
  36. Wajdi Zaghouani, Mona Diab, Aous Mansouri, Sameer Pradhan, and Martha Palmer. 2010. The revised arabic propbank. In Proceedings of the Fourth Linguistic Annotation Workshop, pages 222–226. Association for Computational Linguistics. find