Elamur Research⌕
☰
home / papers / 2013 / Outsourcing FrameNet to the Crowd

Outsourcing FrameNet to the Crowd

Summary

The paper addresses whether full FrameNet annotation, covering both frame choice and frame element (FE) labelling, can be delegated to non-expert crowd workers. Standard tools such as Salto and the Berkeley system use two stages: the annotator picks the frame evoked by a lexical unit (LU), then assigns FEs. The authors argue this is redundant and less cognitively natural. They propose a single-step, bottom-up alternative in which workers match FE definitions to text chunks, including definitions from a competing frame and a None option, so the frame emerges from the FE choices. Experiments ran on CrowdFlower using FrameNet 1.5 expert annotations as gold data. They first reproduced the frame discrimination study of Hong and Baker (2011), then compared both workflows on four ambiguous verbs. Because original FE definitions proved too technical for the crowd, they used manually simplified versions. The one-step method gave higher accuracy and shorter completion time but cost more per sentence. This suggests semantic role annotation can be crowdsourced when task design and definitions are adapted for laypeople.

Key points

  • Reproduction of prior work. Frame discrimination was replicated for the six LUs from Hong and Baker (2011): high.a, history.n, range.n, rip.v, thirst.n and top.a. Accuracies ranged from about 82% to 99% and broadly matched the earlier results, with rip.v the main exception.

  • Comparison setup. The two workflows were compared on four verbal LUs, each evoking two frames: disappear.v, guide.v, heap.v and throw.v.

  • Definition variants. Three kinds of FE definitions were piloted: the original FrameNet wording, a version rewritten in simpler terms by hand, and one simplified by machine using the Heilman and Smith (2010) question generator. The manual simplification worked best.

  • Results.

    WorkflowFinal accuracyTimeCost per question
    2-step.900 (frame discrimination), .687 (after FE recognition)102 h + 69 h = 171 h1.83 ¢ + 2.74 ¢
    1-step.792130 h8.41 ¢ (about 4.6 questions per sentence vs 1.5)

  • Upper bound and weaknesses. Expert agreement reported by Padó and Lapata (2009) served as the upper bound: .897 for frames and .949 for FEs. In the one-step setup, untrusted judgments rose to 217, largely because workers struggled with cross-frame gold units where None was the required answer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semantic role labeling

Cited by

17 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 15 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{fossati2013outsourcing,
  title = {{Outsourcing FrameNet to the Crowd}},
  author = {Fossati, Marco and Giuliano, Claudio and Tonelli, Sara},
  booktitle = {Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2013},
  url = {https://aclanthology.org/P13-2130/}
}

References

14 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Collin F Baker. 2012. Framenet, current collaborations and future goals. Language Resources and Evaluation, pages 1–18. find
  2. Aljoscha Burchardt, Katrin Erk, Anette Frank, Andrea Kowalski, Sebastian Pado, and Manfred Pinkal. 2006. Salto–a versatile multi-level annotation tool. In Proceedings of LREC 2006, pages 517–520. Citeseer. find
  3. Jon Chamberlain, Massimo Poesio, and Udo Kruschwitz. 2008. Phrase detectives: A web-based collaborative annotation game. Proceedings of I-Semantics, Graz. find
  4. Jon Chamberlain, Udo Kruschwitz, and Massimo Poesio. 2009. Constructing an anaphorically annotated corpus with non-experts: Assessing the quality of collaborative annotations. In Proceedings of the 2009 Workshop on The People’s Web Meets NLP: Collaboratively Constructed Semantic Resources, pages 57–… find
  5. Charles J. Fillmore, Collin F. Baker, and Hiroaki Sato. 2002. The FrameNet Database and Software Tools. In Proceedings of the Third International Conference on Language Resources and Evaluation (LREC 2002), pages 1157– 1160, Las Palmas, Spain. find our card
  6. Charles J. Fillmore. 1976. Frame Semantics and the nature of language. In Annals of the New York Academy of Sciences: Conference on the Origin and Development of Language, pages 20–32. Blackwell Publishing. find
  7. Michael Heilman and Noah A. Smith. 2010. Extracting Simplified Statements for Factual Question Generation. In Proceedings of QG2010: The Third Workshop on Question Generation, Pittsburgh, PA, USA. find
  8. Jisup Hong and Collin F Baker. 2011. How good is the crowd at “real” wsd? ACL HLT 2011, page 30. find
  9. Matteo Negri, Luisa Bentivogli, Yashar Mehdad, Danilo Giampiccolo, and Alessandro Marchetti. 2011. Divide and conquer: crowdsourcing the creation of cross-lingual textual entailment corpora. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’11, pages 670–67… find
  10. Sebastian Padó and Mirella Lapata. 2009. Cross-lingual annotation projection for semantic roles. Journal of Artificial Intelligence Research, 36(1):307–340. find
  11. Martha Palmer, Dan Gildea, and Paul Kingsbury. 2005. The Proposition Bank: A Corpus Annotated with Semantic Roles. Computational Linguistics, 31(1). find
  12. Josef Ruppenhofer, Michael Ellsworth, Miriam R.L. Petruck, Christopher R. Johnson, and Jan Scheffczyk. 2006. FrameNet II: Extended Theory and Practice. Available at http://framenet.icsi.berkeley.edu/book/book.html. ↗
  13. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y Ng. 2008. Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 254–263. Association for Computational Lingui… find
  14. Luis Von Ahn, Mihir Kedia, and Manuel Blum. 2006. Verbosity: a game for collecting common-sense facts. In Proceedings of the SIGCHI conference on Human Factors in computing systems, pages 75–78. ACM. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fossati-2013-outsourcing-framenet-crowd/