Elamur Research⌕
☰
home / papers / 2014 / It's All Fun and Games until Someone Annotates: Video Games with a Purpose for Linguistic Annotation

It's All Fun and Games until Someone Annotates: Video Games with a Purpose for Linguistic Annotation

Paper · 2014 · David Jurgens and Roberto Navigli

Summary

Building annotated corpora is slow and expensive. Crowdsourcing lowers the cost, but tasks must be simplified for non-experts, and they can be boring enough to drive workers away. Earlier games with a purpose (GWAPs) for NLP mostly wrapped text-based annotation in scores and leaderboards. Jurgens and Navigli reverse this approach. They start from a playable graphical video game and make the annotation task a core game mechanic, with no linguistic terminology shown to players.

They present two games:

  • Puzzle Racer, a racing-plus-puzzle game in which players pick images that match a theme. This links WordNet senses of nouns, verbs and adjectives, both concrete and abstract, to graded images.
  • Ka-boom!, which performs Word Sense Disambiguation as players interact with pictures.

Puzzle Racer matched expert annotation quality at substantially lower cost than equivalent crowdsourcing. Ka-boom! clearly beat Wordrobe, a state-of-the-art text-based WSD game. The work shows that real video games can serve as a channel for high-quality semantic annotation. It also supplies reusable image-sense resources for further visual and multilingual games.

Key points

  • Puzzle Racer produced an image library of 19,073 images covering 443 WordNet senses across nouns, verbs and adjectives. Its quality equaled expert ratings at a cost at least 73% lower than crowdsourcing the same resource.
  • Ka-boom! improved WSD accuracy by 16.3% over Wordrobe (p < 0.01). Its gain over the MFS baseline was not statistically significant (p = 0.14).
  • Each game yields several sense judgments per context. Ka-boom! surpassed MFS on nouns after only two games, whereas Wordrobe's accuracy fell as more players disagreed.
  • Earlier image libraries cover only concrete nouns:
    • ImageNet has 3.2M images for 5,247 noun synsets.
    • BabelNet has images for 38.6% of WordNet senses.
  • The released resources include sense-image mappings, image word labels, sense-related Web queries and image-based sense annotations (lcl.uniroma1.it/videogames). The game software is planned for open-source release.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sapienza University of RomeRome · Italy57 works · 9,564 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 worksComputer Science Applications · 102 works

Cite

Cite as David Jurgens and Roberto Navigli. 2014. It's All Fun and Games until Someone Annotates: Video Games with a Purpose for Linguistic Annotation. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00195

@article{jurgens2014it,
  title = {{It's All Fun and Games until Someone Annotates: Video Games with a Purpose for Linguistic Annotation}},
  author = {David Jurgens and Roberto Navigli},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2014},
  doi = {10.1162/tacl\_a\_00195},
  url = {https://doi.org/10.1162/tacl_a_00195}
}

References

40 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Guillaume Artignan, Mountaz Hascoët, and Mathieu Lafourcade. 2009. Multiscale visual analysis of lexical networks. In Proceedings of the International Conference on Information Visualisation, pages 685–690. find
  2. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. find
  3. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In Proceedings of the 17th International Conference on Computational Linguistics and 36th Annual Meeting of the Association for Computational Linguistics, Montréal, Québec, Canada, 10–14 August 1998, Montr…
  4. Kobus Barnard, Pinar Duygulu, David Forsyth, Nando De Freitas, David M. Blei, and Michael I. Jordan. 2003. Matching words and pictures. The Journal of Machine Learning Research, 3:1107–1135. find
  5. Chris Biemann and Valerie Nygaard. 2010. Crowdsourcing WordNet. In The 5th International Conference of the Global WordNet Association (GWC-2010). Jon Chamberlain, Karën Fort, Udo Kruschwitz, Mathieu Lafourcade, and Massimo Poesio. 2013. Using games to create language resources: Successes and limita… find
  6. Springer. Tim Chklovski and Yolanda Gil. 2005. Improving the design of intelligent acquisition interfaces for collecting world knowledge from web contributors. In Proceedings of the International Conference on Knowledge Capture, pages 35–42. find
  7. ACM. Tim Chklovski and Rada Mihalcea. 2002. Building a Sense Tagged Corpus with Open Mind Word Expert. In Proceedings of ACL 2002 Workshop on WSD: Recent Successes and Future Directions, Philadelphia, PA, USA. Seth Cooper, Firas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, A… find
  8. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255. find
  9. Philip Edmonds and Adam Kilgarriff. 2002. Introduction to the special issue on evaluating word sense disambiguation systems. Natural Language Engineering, 8(4):279–291. find
  10. Katrin Erk and Diana McCarthy. 2009. Graded word sense assignment. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 440–449, Singapore. find
  11. Katrin Erk, Diana McCarthy, and Nicholas Gaylord. 2012. Measuring word meaning in context. Computational Linguistics, 39(3):511–554.
  12. Christiane Fellbaum, Joachim Grabowski, and Shari Landes. 1998. Performance and confidence in a semantic annotation task. In Christiane Fellbaum, editor, Word-Net: An electronic lexical database, pages 217–237. MIT Press. Christiane Fellbaum, editor. 1998. find
  13. WordNet: An Electronic Database. MIT Press, Cambridge, MA. Yansong Feng and Mirella Lapata. 2013. Automatic caption generation for news images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(4):797–812. find
  14. Tiziano Flati and Roberto Navigli. 2013. SPred: Large-scale Harvesting of Semantic Predicates. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL), pages 1222–1232, Sofia, Bulgaria. Amaç Herdağdelen and Marco Baroni. 2012. Bootstrapping a game with a pu… find
  15. Jisup Hong and Collin F. Baker. 2011. How Good is the Crowd at “real” WSD? In Proceedings of the Fifth Linguistic Annotation Workshop (LAW V), pages 30– 37. find
  16. ACL. Eduard H. Hovy, Roberto Navigli, and Simone Paolo Ponzetto. 2013. Collaboratively built semi-structured content and Artificial Intelligence: The story so far. Artificial Intelligence, 194:2–27. find
  17. David Jurgens. 2013. Embracing ambiguity: A comparison of annotation methodologies for crowdsourcing word sense labels. In Proceedings of the Conference of the North American Chapter of the Association of Computational Linguistics (NAACL), pages 556–562. find
  18. Ramesh Krishnamurthy and Diane Nicholls. 2000. Peeling an onion: The lexicographer’s experience of manual sense-tagging. Computers and the Humanities, 34(1-2):85–97. find
  19. Girish Kulkarni, Visruth Premraj, Vicente Ordonez, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C. Berg, and Tamara L. Berg. 2013. Babytalk: Understanding and generating simple image descriptions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(12):2891–2903. find
  20. Yen-ling Kuo, Jong-Chuan Lee, Kai-yang Chiang, Rex Wang, Edward Shen, Cheng-wei Chan, and Jane Yung-jen Hsu. 2009. Community-based game design: experiments on social games for commonsense data collection. In Proceedings of the ACM SIGKDD Workshop on Human Computation, pages 15–22. find
  21. Mathieu Lafourcade and Alain Joubert. 2010. Computing trees of named word usages from a crowdsourced lexical network. In Proceedings of the International Multiconference on Computer Science and Information Technology (IMCSIT), pages 439–446, Wisla, Poland. find
  22. Tak Yeon Lee, Casey Dugan, Werner Geyer, Tristan Ratchford, Jamie Rasmussen, N. Sadat Shami, and Stela Lupushor. 2013. Experiments on motivational feedback for crowdsourced workers. In Seventh International AAAI Conference on Weblogs and Social Media (ICWSM), pages 341–350. find
  23. Thomas Mensink, Jakob J. Verbeek, and Gabriela Csurka. 2013. Tree-structured crf models for interactive image labeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(2):476–489. find
  24. George A. Miller, Claudia Leacock, Randee Tengi, and Ross Bunker. 1993. A semantic concordance. In Proceedings of the 3rd DARPA Workshop on Human Language Technology, pages 303–308, Plainsboro, N.J.
  25. Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence, 193:217–250. find
  26. Roberto Navigli. 2009. Word Sense Disambiguation: A survey. ACM Computing Surveys (CSUR), 41(2):1–69. find
  27. Martha Palmer, Hoa Dang, and Christiane Fellbaum. 2007. Making fine-grained and coarse-grained sense distinctions, both manually and automatically. Natural Language Engineering, 13(2):137–163. find
  28. Rebecca J. Passonneau and Bob Carpenter. 2013. The benefits of a model of annotation. In 7th Linguistic Annotation Workshop and Interoperability with Discourse, August 8–9.
  29. Rebecca J. Passonneau, Vikas Bhardwaj, Ansaf Salleb-Aouissi, and Nancy Ide. 2012. Multiplicity and word sense: evaluating and learning from multiply labeled word sense annotations. Language Resources and Evaluation, 46(2):209–252. find
  30. Ei Pa Pa Pe-Than, DH-L Goh, and Chei Sian Lee. 2012. A survey and typology of human computation games. In Information Technology: New Generations (ITNG), 2012 Ninth International Conference on, pages 720– 725. find
  31. IEEE. Mohammad Taher Pilehvar and Roberto Navigli. 2014. A Large-scale Pseudoword-based Evaluation Framework for State-of-the-Art Word Sense Disambiguation. Computational Linguistics, 40(4).
  32. Massimo Poesio, Jon Chamberlain, Udo Kruschwitz, Livio Robaldo, and Luca Ducceschi. 2013. Phrase detectives: Utilizing collective intelligence for internet-scale language resource creation. ACM Transactions on Interactive Intelligent Systems, 3(1):3:1–3:44, April. find
  33. Nitin Seemakurty, Jonathan Chu, Luis Von Ahn, and Anthony Tomasic. 2010. Word sense disambiguation via human computation. In Proceedings of the ACM SIGKDD Workshop on Human Computation, pages 60–63. find
  34. ACM. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Ng. 2008. Cheap and fast – but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, Waikiki, Honolulu, Hawaii, 25-27 October,…
  35. Antonio Torralba, Robert Fergus, and William T Freeman. 2008. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(11):1958–1970. find
  36. Daniele Vannella, David Jurgens, Daniele Scarfini, Domenico Toscani, and Roberto Navigli. 2014. Validating and extending semantic knowledge bases using video games with a purpose. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 1294–1304. find
  37. Noortje J. Venhuizen, Valerio Basile, Kilian Evang, and Johan Bos. 2013. Gamification for word sense labeling. In Proceedings of the International Conference on Computational Semantics (IWCS), pages 397–403. find
  38. Luis von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In Proceedings of the Conference on Human Factors in Computing Systems (CHI), pages 319–326. find
  39. Luis von Ahn and Laura Dabbish. 2008. Designing games with a purpose. Communications of the ACM, 51(8):58–67. find
  40. Aobo Wang, Cong Duy Vu Hoang, and Min-Yen Kan. 2013. Perspectives on crowdsourcing annotations for natural language processing. Language Resources and Evaluation, 47(1):9–31. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jurgens-2014-it-s-all-fun-games/