Elamur Research⌕
☰
home / papers / 2022 / Design Choices in Crowdsourcing Discourse Relation Annotations: The Effect of Worker Selection and Training

Design Choices in Crowdsourcing Discourse Relation Annotations: The Effect of Worker Selection and Training

Summary

Getting lay crowdworkers to label implicit discourse relations is difficult. The task depends on interpretation, and PDTB 3.0 distinguishes 36 sense types that novices cannot realistically master. This paper asks how recruiting and preparing workers affects label quality, measured as agreement between worker labels and gold labels. Two existing natural-language annotation methods are tested in parallel:

  • the two-step discourse connective (DC) insertion task of Yung et al. (2019);
  • the question-answer (QA) approach behind QADiscourse, which the authors adapt to relations spanning two sentences.

Three studies on inter-sentential implicit relations from Wikipedia and the Blog Authorship Corpus compare unscreened, untrained workers (Study 1), screened and trained workers (Study 2), and screened but untrained workers (Study 3). Unscreened workers left much room for improvement. Screening plus training raised quality but cost considerable time and money, partly because workers dropped out. Screening alone gave labels of roughly the same quality as training did. Because this held for both methods, selection-only recruitment appears to be a cheaper, practical option for other crowdsourced discourse annotation projects.

Key points

  • Target scheme is Penn Discourse Treebank 3.0 (PDTB), which defines 36 relation senses. Both crowdsourcing methods map worker output onto PDTB3 labels.
  • DC method: workers first type a connective freely, then choose from a disambiguating list derived from a connective bank. If the first answer is empty or unmatched, they get a default list of 12 connectives.
  • QA method: builds on QADiscourse (over 16,600 QA pairs, 17 question prefixes, intra-sentential relations). It was extended here with more prefixes, a refined PDTB mapping, and sentence-level question/answer marking.
  • Worker numbers: Study 1 had 20 workers. Study 2 had 10 (DC) and 18 (QA). Study 3 had 30 per method. Workers were mostly based in the UK, with some from Canada, the US and Ireland.
  • In Study 2, workers agreed well on synthetic items, showing that both methods let workers express the intended relation. Remaining disagreement is attributed to the inherent difficulty of inferring implicit relations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

9 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{scholman2022design,
  title = {{Design Choices in Crowdsourcing Discourse Relation Annotations: The Effect of Worker Selection and Training}},
  author = {Scholman, Merel and Pyatkin, Valentina and Yung, Frances and Dagan, Ido and Tsarfaty, Reut and Demberg, Vera},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  url = {https://aclanthology.org/2022.lrec-1.231/}
}

References

19 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alagarai Sampath, H., Rajeshuni, R., and Indurkhya, B. (2014). Cognitively inspired task design to improve user performance on crowdsourcing platforms. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 3665– 3674. find
  2. Aroyo, L. and Welty, C. (2013). Crowd truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard. WebSci2013 ACM, 2013(2013). find
  3. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. find
  4. Finnerty, A., Kucherbaev, P., Tranquillini, S., and Con-vertino, G. (2013). Keep it simple: Reward and task design in crowdsourcing. In Proceedings of the Biannual Conference of the Italian Chapter of SIGCHI, pages 1–4. find
  5. Hirth, M., Hoßfeld, T., and Tran-Gia, P. (2013). Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms. Mathematical and Computer Modelling, 57(11-12):2918–2932. find
  6. Hoek, J., Scholman, M. C., and Sanders, T. J. (2021). Is there less annotator agreement when the discourse relation is underspecified? In Integrating Perspectives on Discourse Annotation (DiscAnn), pages 1– 6. find our card
  7. Kawahara, D., Machida, Y., Shibata, T., Kurohashi, S., Kobayashi, H., and Sassano, M. (2014). Rapid development of a corpus with discourse annotations using two-stage crowdsourcing. In COLING. find
  8. Kishimoto, Y., Sawada, S., Murawaki, Y., Kawahara, D., and Kurohashi, S. (2018). Improving crowdsourcing-based annotation of japanese discourse relations. In LREC. find our card
  9. Lee, J.-U., Klie, J.-C., and Gurevych, I. (2021). Annotation curricula to implicitly train non-expert annotators. ArXiv, abs/2106.02382. find
  10. Nangia, N., Sugawara, S., Trivedi, H., Warstadt, A., Vania, C., and Bowman, S. (2021). What ingredients make for an effective crowdsourcing protocol for difficult nlu data collection tasks? In ACL/IJCNLP. Parrish, A., Huang, W., Agha, O., hwan Lee, S., Nangia, N., Warstadt, A., Aggarwal, K., Allaway… find
  11. Pyatkin, V., Klein, A., Tsarfaty, R., and Dagan, I. (2020). QADiscourse-Discourse Relations as QA Pairs: Representation, crowdsourcing and baselines. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2804–2819. find our card
  12. Rechkemmer, A. and Yin, M. (2020). Motivating novice crowd workers through goal setting: An investigation into the effects on complex crowdsourcing task training. In AAAI 2020. find
  13. Roit, P., Klein, A., Stepanov, D., Mamou, J., Michael, J., Stanovsky, G., Zettlemoyer, L., and Dagan, I. (2020). Controlled crowdsourcing for high-quality qa-srl annotation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7008–7013. find our card
  14. Schler, J., Koppel, M., Argamon, S., and Pennebaker, J. W. (2006). Effects of age and gender on blogging. In AAAI spring symposium: Computational approaches to analyzing weblogs, volume 6, pages 199–205. find
  15. Snow, R., O’Connor, B., Jurafsky, D., and Ng, A. Y. (2008). Cheap and fast—but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 254–263, Waikiki, HI. find
  16. Spooren, W. P. M. S. and Degand, L. (2010). Coding coherence relations: Reliability and validity. Corpus Linguistics and Linguistic Theory, 6(2):241–266. find
  17. Tauchmann, C., Daxenberger, J., and Mieskes, M. (2020). The influence of input data complexity on crowdsourcing quality. In Proceedings of the 25th International Conference on Intelligent User Interfaces Companion, pages 71–72. find
  18. Webber, B., Prasad, R., Lee, A., and Joshi, A. (2019). The Penn Discourse Treebank 3.0 annotation manual. Philadelphia, University of Pennsylvania. find
  19. Yung, F., Demberg, V., and Scholman, M. (2019). Crowdsourcing discourse relation annotations by a two-step connective insertion task. In Proceedings of the 13th Linguistic Annotation Workshop, pages 16–25. find our card

↑ back to top