Elamur Research⌕
☰
home / papers / 2008 / Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks

Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks

Summary

Expert linguistic annotation is slow and costly. This paper asks whether crowds of paid non-experts on Amazon Mechanical Turk (AMT) can stand in for specialists. The authors picked five tasks for which expert gold labels already existed: judging emotion in text (affect recognition), rating how similar pairs of words are, deciding textual entailment (RTE), placing events in temporal order, and word sense disambiguation. They used short task descriptions with an example for each class and restricted responses to multiple-choice or bounded numeric input. They gathered ten independent annotations per item, which let them measure how label quality grows as more workers are added. On all five tasks, crowd labels agreed well with expert labels. A small number of averaged non-expert judgments typically matched one expert. For affect recognition, a classifier trained on crowd labels performed as well as, or better than, one trained on a single expert's labels. The authors also introduced a bias-correction scheme that weights workers using a small set of gold-labelled items. The work matters because it showed that large annotation projects can be run at a small fraction of the usual price, and it gave a practical recipe for checking quality.

Key points

  • Affect data: 100 headlines were sampled from the SemEval affective text test set. Each was rated for six emotions plus valence, with 10 annotations per label type, for 7000 labels in total. The whole collection cost US$2.00, which is 3500 labels per dollar or at least 875 expert-equivalent labels per dollar.
  • Workers needed per expert: On average, about 4 non-expert labels per item matched the agreement of one expert. For most emotions, expert level was reached with 9 or fewer workers, often with just 2. "Fear" was the exception and never reached expert level.
  • Better gold labels: Adding crowd labels to the expert pool raised agreement for experts. Average Pearson correlation across all tasks went from 0.580 to 0.607.
  • Bias correction: Per-worker models were calibrated on gold items, using Laplace smoothing with 1 pseudocount, uniform priors, and 20-fold cross-validation. This beat naive majority voting on RTE-1 (about +4.0% accuracy) and on event annotation. Word sense disambiguation was already at maximum accuracy, so it could not improve.
  • Training a classifier: A simple bag-of-words unigram affect model similar to SWAT was trained on 100 headlines and tested on 900. For five of the seven label types, a model trained on one set of crowd labels beat the average model trained on one expert's labels.

Interesting ideas

  • I1. Express crowd cost in expert units. Convert cost into "expert-equivalent labels per dollar" by finding how many cheap labels match one expert. This makes annotation budgets comparable across vendors or labelling strategies, for example when planning a Ukrainian corpus.
  • I2. Collect deliberate redundancy. Gather many labels per item, such as ten, then resample subsets of size k to plot quality against annotator count. This tells you the cheapest redundancy level for production before you commit the full budget.
  • I3. Calibrate workers on a gold subset. Seed a few expert-labelled items and use them to estimate each worker's reliability, then use these estimates to weight votes. This is a cheap way to beat plain majority voting and applies to any categorical labelling job.
  • I4. Mix many annotators to reduce bias. A label set pooled from several people can train a better model than one consistent expert, possibly because individual biases average out. When hiring annotators, it may be worth spreading items across several people rather than relying on one.
  • I5. Score annotators with leave-one-out agreement. Correlate one annotator with the average of the others, and compare expert-vs-expert with crowd-vs-expert scores. This gives a fair reference point for numeric rating tasks where no single "true" label exists.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Word sense disambiguation

Cited by

390 papers of our library cite it (of the 140884 whose bibliographies we read); 40 of them have a page here, 350 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{snow2008cheap,
  title = {{Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks}},
  author = {Snow, Rion and O'Connor, Brendan and Jurafsky, Daniel and Ng, Andrew},
  booktitle = {Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing},
  year = {2008},
  url = {https://aclanthology.org/D08-1027/}
}

References

33 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Paul S. Albert and Lori E. Dodd. 2004. A Cautionary Note on the Robustness of Latent Class Models for Estimating Diagnostic Error without a Gold Standard. Biometrics, Vol. 60 (2004), pp. 427-435. find
  2. Collin F. Baker, Charles J. Fillmore and John B. Lowe. 1998. The Berkeley FrameNet project. In Proc. of COLING-ACL 1998. find our card
  3. Michele Banko and Eric Brill. 2001. Scaling to Very Very Large Corpora for Natural Language Disambiguation. In Proc. of ACL-2001. find our card
  4. Junfu Cai, Wee Sun Lee and Yee Whye Teh. 2007. Improving Word Sense Disambiguation Using Topic Features. In Proc. of EMNLP-2007 . find
  5. Timothy Chklovski and Rada Mihalcea. 2002. Building a sense tagged corpus with Open Mind Word Expert. In Proc. of the Workshop on ”Word Sense Disambiguation: Recent Successes and Future Directions”, ACL 2002. find
  6. Timothy Chklovski and Yolanda Gil. 2005. Towards Managing Knowledge Collection from Volunteer Contributors. Proceedings of AAAI Spring Symposium on Knowledge Collection from Volunteer Contributors (KCVC05). find
  7. Ido Dagan, Oren Glickman and Bernardo Magnini. 2006. The PASCAL Recognising Textual Entailment Challenge. Machine Learning Challenges. Lecture Notes in Computer Science, Vol. 3944, pp. 177-190, Springer, 2006. find
  8. Wisam Dakka and Panagiotis G. Ipeirotis. 2008. Automatic Extraction of Useful Facet Terms from Text Documents. In Proc. of ICDE-2008. find
  9. A. P. Dawid and A. M. Skene. 1979. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Applied Statistics, Vol. 28, No. 1 (1979), pp. 20-28. find
  10. Michael Kaisser and John B. Lowe. 2008. A Research Collection of QuestionAnswer Sentence Pairs. In Proc. of LREC-2008. find
  11. Michael Kaisser, Marti Hearst, and John B. Lowe. 2008. Evidence for Varying Search Results Summary Lengths. In Proc. of ACL-2008. find
  12. Phil Katz, Matthew Singleton, Richard Wicentowski. 2007. SWAT-MP: The SemEval-2007 Systems for Task 5 and Task 14. In Proc. of SemEval-2007. find
  13. Aniket Kittur, Ed H. Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with Mechanical Turk. In Proc. of CHI-2008. find
  14. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics 19:2, June 1993. find our card
  15. George A. Miller and William G. Charles. 1991. Contextual Correlates of Semantic Similarity. Language and Cognitive Processes, vol. 6, no. 1, pp. 1-28, 1991. find
  16. George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunke. 1993. A semantic concordance. In Proc. of HLT-1993. find
  17. Preslav Nakov. 2008. Paraphrasing Verbs for Noun Compound Interpretation. In Proc. of the Workshop on Multiword Expressions, LREC-2008. find
  18. Martha Palmer, Dan Gildea, and Paul Kingsbury. 2005. The Proposition Bank: A Corpus Annotated with Semantic Roles. Computational Linguistics, 31:1. find
  19. Sameer Pradhan, Edward Loper, Dmitriy Dligach and Martha Palmer. 2007. SemEval-2007 Task-17: English Lexical Sample, SRL and All Words. In Proc. of SemEval-2007 . find our card
  20. James Pustejovsky, Patrick Hanks, Roser Saur, Andrew See, Robert Gaizauskas, Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro and Marcia Lazo. 2003. The TIMEBANK Corpus. In Proc. of Corpus Linguistics 2003, 647-656. find
  21. Philip Resnik. 1999. Semantic Similarity in a Taxonomy: An Information-Based Measure and its Application to Problems of Ambiguity in Natural Language. JAIR, Volume 11, pages 95-130. find
  22. Herbert Rubenstein and John B. Goodenough. 1965. Contextual Correlates of Synonymy. Communications of the ACM, 8(10):627–633. find
  23. Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers. In Proc. of KDD-2008. find
  24. Push Singh. 2002. The public acquisition of commonsense knowledge. In Proc. of AAAI Spring Symposium on Acquiring (and Using) Linguistic (and World) Knowledge for Information Access, 2002. find
  25. Alexander Sorokin and David Forsyth. 2008. Utility data annotation with Amazon Mechanical Turk. To appear in Proc. of First IEEE Workshop on Internet Vision at CVPR, 2008. See also: http://vision.cs.uiuc.edu/annotation/ ↗
  26. David G. Stork. 1999. The Open Mind Initiative. IEEE Expert Systems and Their Applications pp. 16-20, May/June 1999. find
  27. Carlo Strapparava and Rada Mihalcea. 2007. SemEval-2007 Task 14: Affective Text In Proc. of SemEval-2007. find our card
  28. Qi Su, Dmitry Pavlov, Jyh-Herng Chow, and Wendell C. Baker. 2007. Internet-Scale Collection of Human-Reviewed Data. In Proc. of WWW-2007. find
  29. Luis von Ahn and Laura Dabbish. 2004. Labeling Images with a Computer Game. In ACM Conference on Human Factors in Computing Systems, CHI 2004. find
  30. Luis von Ahn, Mihir Kedia and Manuel Blum. 2006. Verbosity: A Game for Collecting Common-Sense Knowledge. In ACM Conference on Human Factors in Computing Systems, CHI Notes 2006. find
  31. Ellen Voorhees and Hoa Trang Dang. 2006. Overview of the TREC 2005 question answering track. In Proc. of TREC-2005. find
  32. Janyce M. Wiebe, Rebecca F. Bruce and Thomas P. O’Hara. 1999. Development and use of a gold-standard data set for subjectivity classifications. In Proc. of ACL-1999. find
  33. Annie Zaenen. Submitted. Do give a penny for their thoughts. International Journal of Natural Language Engineering (submitted). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/snow-2008-cheap-fast-but-is-it/