Elamur Research
☰
home / papers / 2013 / Learning Whom to Trust with MACE

Learning Whom to Trust with MACE

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/N13-1132.pdf

Summary

Crowdsourcing platforms such as Amazon's Mechanical Turk (AMT) make it cheap to collect categorical labels for NLP. However, some workers answer carelessly or strategically to earn money quickly, for example by always choosing the same option. Majority voting can be misled by such spammers. The authors propose MACE (Multi-Annotator Competence Estimation), an unsupervised item-response model. It treats true labels as hidden variables and gives each annotator a trustworthiness parameter plus a separate label distribution used only when that person is spamming. When not spamming, an annotator is assumed to give the correct label. The model is trained with EM or with Variational Bayes using sparse priors. They evaluate it on AMT datasets from Snow et al. (2008) and on synthetic adversarial data. MACE beats majority voting, matches more complex state-of-the-art systems, and estimates annotator reliability far better than agreement-based measures. Confidence thresholding and a few gold control items raise accuracy further. This matters because it allows cleaner corpora to be built cheaply from noisy crowd labels and helps identify which workers to rehire or block.

Key points

  • Data: three discrete-label tasks from Snow et al. (2008):
    • WSD: 177 items, 34 annotators.
    • RTE: 800 items, 164 annotators.
    • Temporal: 462 items, 76 annotators.
    • Synthetic sets: 100 items with 2 or 4 labels, 20 simulated annotators, 10 answers per item.
  • Training setup:
    • EM runs 50 iterations with 100 random restarts and light smoothing of the M-step.
    • The VB variant uses a symmetric Beta(0.5, 0.5) prior on trustworthiness and a Dirichlet prior fixed at 10.0 on spamming strategies.
  • Annotator reliability: MACE-VB's trust estimates correlate with true annotator proficiency at Pearson ρ ≈ 0.90–0.91 on all three datasets. Raw agreement reaches 0.73–0.81, and Cohen's κ falls as low as 0.13 on WSD.
  • Extra accuracy gains:
    • Keeping only low posterior-entropy predictions trades coverage for higher accuracy.
    • Supplying gold labels for some items as token constraints helps, and MACE exploits small amounts of supervision more effectively than the baseline.
  • Release: MACE is a freely downloadable Java implementation from ISI. Its accuracy is comparable to the more complex model of Raykar and Yu (2012).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

214 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 208 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{hovy2013learning,
  title = {{Learning Whom to Trust with MACE}},
  author = {Hovy, Dirk and Berg-Kirkpatrick, Taylor and Vaswani, Ashish and Hovy, Eduard},
  booktitle = {Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2013},
  url = {https://aclanthology.org/N13-1132/}
}

References

23 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jiang Bian, Yandong Liu, Ding Zhou, Eugene Agichtein, and Hongyuan Zha. 2009. Learning to recognize reliable users and content in social media with coupled mutual reinforcement. In Proceedings of the 18th international conference on World wide web, pages 51– 60. ACM. find
  2. Chris Callison-Burch and Mark Dredze. 2010. Creating speech and language data with amazon’s mechanical turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pages 1–12, Los Angeles, June. Association for Computational Linguistics. find
  3. Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar Zaidan. 2010. Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and Metri… find
  4. Bob Carpenter. 2008. Multilevel Bayesian models of categorical data annotation. Unpublished manuscript. find
  5. A. Philip Dawid and Allan M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, pages 20–28. find
  6. Arthur P. Dempster, Nan M. Laird, and Donald B. Rubin. 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 39(1):1–38. find
  7. Jason Eisner. 2002. An interactive spreadsheet for teaching the forward-backward algorithm. In Proceedings of the ACL-02 Workshop on Effective tools and methodologies for teaching natural language processing and computational linguistics-Volume 1, pages 10– 18. Association for Computational Linguist… find
  8. Alvan R. Feinstein and Domenic V. Cicchetti. 1990. High agreement but low kappa: I. the problems of two paradoxes. Journal of Clinical Epidemiology, 43(6):543–549. find
  9. Kilem Li Gwet. 2008. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29–48. find
  10. Eduard Hovy. 2010. Annotation. A Tutorial. In 48th Annual Meeting of the Association for Computational Linguistics. find
  11. Mukund Jha, Jacob Andreas, Kapil Thadani, Sara Rosenthal, and Kathleen McKeown. 2010. Corpus creation for new genres: A crowdsourced approach to pp attachment. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pages 13–20. Association f… find
  12. Mark Johnson. 2007. Why doesn’t EM find good HMM POS-taggers. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 296–305. find
  13. Michael D. Lee, Mark Steyvers, Mindy de Young, and Brent J. Miller. 2011. A model-based approach to measuring expertise in ranking tasks. In L. Carlson, C. Hölscher, and T.F. Shipley, editors, Proceedings of the 33rd Annual Conference of the Cognitive Science Society, Austin, TX. Cognitive Science… find
  14. Vikas C. Raykar and Shipeng Yu. 2012. Eliminating Spammers and Ranking Annotators for Crowdsourced Labeling Tasks. Journal of Machine Learning Research, 13:491–518. find
  15. Padhraic Smyth, Usama Fayyad, Mike Burl, Pietro Perona, and Pierre Baldi. 1995. Inferring ground truth from subjective labelling of Venus images. Advances in neural information processing systems, pages 1085– 1092. find
  16. Rion Snow, Brendan O’Connor, Dan Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast—but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 254–263. Association for Computational Linguisti… find
  17. Alexander Sorokin and David Forsyth. 2008. Utility data annotation with Amazon Mechanical Turk. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPRW ’08, pages 1–8. IEEE. find
  18. Mark Steyvers, Michael D. Lee, Brent Miller, and Pernille Hemmer. 2009. The wisdom of crowds in the recollection of order information. Advances in neural information processing systems, 23. find
  19. Stephen Tratz and Eduard Hovy. 2010. A taxonomy, dataset, and classifier for automatic noun compound interpretation. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 678–687. Association for Computational Linguistics. find
  20. Peter Welinder, Steve Branson, Serge Belongie, and Pietro Perona. 2010. The multidimensional wisdom of crowds. In Neural Information Processing Systems Conference (NIPS), volume 6. find
  21. Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier Movellan. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in Neural Information Processing Systems, 22:2035–2043. find
  22. Yan Yan, Rómer Rosales, Glenn Fung, Mark Schmidt, Gerardo Hermosillo, Luca Bogoni, Linda Moy, and Jennifer Dy. 2010. Modeling annotator expertise: Learning when everybody knows a bit of something. In International Conference on Artificial Intelligence and Statistics. find
  23. Omar F. Zaidan and Chris Callison-Burch. 2011. Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1220– 1229, Portland, Oregon, USA, June. Association f… find