Elamur Research
☰
home / papers / 2015 / Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation

Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation

Written for the Research library (claude-opus-5-5) from the paper text. Original — ojs.aaai.org/aimagazine/index.php/aimagazine/article/download/2564/2468

Summary

Gold standards for training and testing semantic interpretation systems usually assume that each example has exactly one correct label, and they use interannotator agreement as the main quality check. Aroyo and Welty argue that this assumption fails in big data semantics. They name seven common beliefs about annotation: one truth, disagreement as a defect, the value of detailed guidelines, a single annotator being sufficient, expert superiority, equal weight for all examples, and permanent validity of collected data. They challenge each belief using their crowdsourcing experiments on medical relation extraction with UMLS relations. As a replacement they propose crowd truth. Many workers label the same sentence, and the spread of their answers is kept as evidence about how ambiguous a sentence is, how reliable a worker is, and how confusable relations are. These ideas become sentence-, worker- and relation-level measures, linked to the semiotic triangle of reference. The authors report that the approach is cheaper, faster and more scalable than dedicated annotators while improving some quality aspects. It can be transferred to other tasks once a compact answer space is defined.

Key points

  • Over-specified guidelines can mask real ambiguity. The ACE 2002 RDC guidelines V2.3 treat geography as static and so force a located reading on the Monica Lewinsky sentence. Those guidelines took more than a year to build.
  • In standard practice, often over 90% of examples are labeled by just one person. Only a small overlap is used to measure agreement.
  • The crowd experiments covered 16 medical relations plus NONE and OTHER. They used 30 sentences, with 10, 20 and 30 workers per sentence.
  • Crowd and SME accuracy are similar for several relations:
    • TREATS: .81 (crowd) vs .88 (SME)
    • PREVENTS: .88 vs .84
    • DIAGNOSE: .72 vs .89
    • CAUSES: .69 vs .70
    • LOCATION: .83 vs .7
  • Relation metrics:
    • Similarity is the pairwise conditional co-annotation probability.
    • Ambiguity is a relation's maximum similarity to any other relation.
    • Clarity is its maximum sentence-relation score.
    • Example: UMLS MANIFESTATION has a precise definition but was never found in medical text.
    • For event processing (Inel et al. 2013), the method was adapted by mapping disagreement onto a continent/country/city hierarchy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

141 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 131 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1609/aimag.v36i1.2564

@article{aroyo2015truth,
  title = {{Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation}},
  author = {Aroyo, Lora M. and Welty, Chris},
  journal = {AI Magazine},
  year = {2015},
  doi = {10.1609/aimag.v36i1.2564},
  url = {https://doi.org/10.1609/aimag.v36i1.2564}
}

References

22 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alonso, O., and Baeza-Yates, R. 2011. Design and Implementation of Relevance Assessments Using Crowdsourcing. In Advances in Information Retrieval. Lecture Notes in Computer Science Volume 6611, 153–164. Berlin: Springer-Verlag. dx.doi.org/10.1007/978-3-642-20161-5\_16 — [doi:10.1007/978-3-642-20161-5\_16](https://doi.org/10.1007/978-3-642-20161-5_16) find
  2. Aroyo, L., and Welty, C. 2013a. Crowd Truth: Harnessing Disagreement in Crowdsourcing a Relation Extraction Gold Standard. In Web Science 2013. New York: Association for Computing Machinery. find
  3. Aroyo, L., and Welty, C. 2013b. Measuring Crowd Truth for Medical Relation Extraction. In Semantics for Big Data: Papers from the AAAI Fall Symposium, ed. F. van Harmelen, J. A. Hendler, P. Hitzler, and K. Janowicz. AAAI Technical Report FS-13-04, Palo Alto, CA. find
  4. Bozzon, A.; Brambilla, M.; Ceri, S.; and Mauri, A. 2013. Reactive Crowdsourcing. In Proceedings of the 22nd International Conference on World Wide Web, 153–164. New York: Association for Computing Machinery. find
  5. Carlson, A.; Betteridge, J.; Hruschka, Jr., E. R.; and Mitchell, T. M. 2009. Coupling Semi-Supervised Learning of Categories and Relations. In Proceedings of the North American Chapter of the Association for Computation Linguistics Human Language Technologies 2009 Workshop on Semi-Supervised Learnin… find
  6. Chklovski, T., and Mihalcea, R. 2003. Exploiting Agreement and Disagreement of Human Annotators for Word Sense Disambiguation. Paper presented at the International Conference on Recent Advances in Natural Language Processing, 10–12 September 2003. Available from University of North Texas Scholarly W… find
  7. Cohen, J. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20(1): 37– 46. dx.doi.org/10.1177/001316446002000104 — [doi:10.1177/001316446002000104](https://doi.org/10.1177/001316446002000104) find
  8. Difallah, D. E.; Demartini, G.; and Cudré-Mauroux, P. 2012. Mechanical Cheat: Spamming Schemes and Adversarial Techniques on Crowdsourcing Platforms. In Crowdsourcing the Semantic Web: Proceedings of the 1st International Workshop on Crowdsourcing the Semantic Web. CEUR Workshop Proceedings Volume 8… find
  9. Dumitrache, A.; Aroyo, L.; Welty, C.; Sips, R.-J.; and Levas, A. 2013. “Dr. Detective”: Combining Gamification Techniques and Crowdsourcing to Create a Gold Standard in Medical Text. In Crowdsourcing the Semantic Web: Proceedings of the 1st International Workshop on Crowdsourcing the Semantic Web. C… find
  10. Gligorov, R.; Hildebrand, M.; van Ossenbruggen, J.; Schreiber, G.; and Aroyo, L. 2011. On the Role of User-Generated Metadata in Audio Visual Collections. In Proceedings of the 6th International Conference on Knowledge Capture (K-CAP 2011), 145–152. New York: Association for Computing Machinery. find
  11. Inel, O.; Aroyo, L.; Welty, C.; and Sips, R.-J. 2013. Exploiting Crowdsourcing Disagreement with Various Domain-Independent Quality Measures. Paper presented at the 3rd International Workshop on Detection, Representation, and $QLQWHUQDWLRQDOIRUXPRQGHFODUDWLYHORJLFSURJUDPPLQJQRQ PRQRWRQLFUH… find
  12. Ipeirotis, P. G.; Provost, F.; and Wang, J. 2010. Quality Management on Amazon Mechanical Turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation, 64–67. New York: Association for Computing Machinery. dx.doi.org/10.1145/1837885.1837906 — [doi:10.1145/1837885.1837906](https://doi.org/10.1145/1837885.1837906) find
  13. Kittur, A.; Chi, E. H.; and Suh, B. 2008. Crowdsourcing User Studies with Mechanical Turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 453–456. New York: Association for Computing Machinery. find
  14. Leason, T. 2009. Steve: The Art Museum Social Tagging Project: A Report on the Tag Contributor Experience. In Museums and the Web 2009. Silver Spring, MD: Museums and the Web LLC. find
  15. Lee, J. H., and Hu, X. 2012. Generating Ground Truth for Music Mood Classification Using Mechanical Turk. In Proceedings of the 12th ACM/IEEE-CS Joint Conference on Digital Libraries, 129–138. New York: Association for Computing Machinery. dx.doi.org/10.1145/2232817.2232842 — [doi:10.1145/2232817.2232842](https://doi.org/10.1145/2232817.2232842) find
  16. Markines, B.; Cattuto, C.; Menczer, F.; Benz, D.; Hotho, A.; and Stumme, G. 2009. Evaluating Similarity Measures for Emergent Semantics of Social Tagging. In Proceedings of the 18th International Conference on World Wide Web. New York: Association for Computing Machinery. dx.doi.org/10.1145/1526709.… — [doi:10.1145/1526709.1526796](https://doi.org/10.1145/1526709.1526796) find
  17. Mason, W., and Suri, S. 2012. Conducting Behavioral Research on Amazon’s Mechanical Turk. Behavior Research Methods 44(1): 1–23. dx.doi.org/10.3758/s13428-011-0124-6 Ogden, C. K., and Richards, I. 1923. The Meaning of Meaning. — [doi:10.3758/s13428-011-0124-6](https://doi.org/10.3758/s13428-011-0124-6) find
  18. London: Trubner & Co. Plank, B.; Hovy, D.; and Sgaard, A. 2014. Learning Part-of-Speech Taggers with Inter-Annotator Agreement Loss. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL-2014). Stroudsburg, PA: Association for Computatio… find
  19. Raykar, V. C., and Yu, S. 2012. Eliminating Spammers and Ranking Annotators for Crowdsourced Labeling Tasks. Journal of Machine Learning Research 13: 491–518. find
  20. Sheng, V. S.; Provost, F.; and Ipeirotis, P. G. 2008. Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 614–622. New York: Association for Computing Machiner… — [doi:10.1145/1401890.1401965](https://doi.org/10.1145/1401890.1401965) find
  21. Soberon, G.; Aroyo, L.; Welty, C.; Inel, O.; Overmeen, M.; and Lin, H. 2013. Content and Behaviour Based Metrics for Crowd Truth. In Crowdsourcing the Semantic Web: Proceedings of the 1st International Workshop on Crowdsourcing the Semantic Web. CEUR Workshop Proceedings Volume 1030, 56–69. Aachen,… find
  22. Chris Welty is a research scientist at Google Research in New York, having left IBM’s Watson Group in the summer of 2014. He holds a Ph.D. in computer science from Rensselaer Polytechnic Institute. Welty was a member of the technical leadership team for Watson and IBM’s Jeopardy! Challenge and is a… find