Elamur Research⌕
☰
home / papers / 2022 / Coreference Annotation of an Arabic Corpus using a Virtual World Game

Coreference Annotation of an Arabic Corpus using a Virtual World Game

Summary

Arabic coreference resources are small compared with those for English or Chinese. This limits how well resolvers can be trained. The authors present Stroll with a Scroll, a Game-With-A-Purpose (GWAP) that gathers Arabic coreference judgments from players. Unlike earlier coreference games, which were puzzle-style, it places the task inside a 3D virtual world. The player's avatar walks through an old desert town, finds chests and opens scrolls whose torn words must be guessed. After this motivating part, the player answers annotation questions. Documents are pre-processed by the neural Arabic coreference resolver of Aloraini et al. (2020), which supplies mentions and candidate links. Players then classify mentions or validate links proposed by the model or by other players. Answers are combined with the Mention Pair Annotation aggregation method. The paper describes a system design only and reports no collected data or accuracy figures. Its value lies in opening a gamified route to larger Arabic coreference corpora and in testing whether a virtual world attracts more players than puzzles do.

Key points

  • Motivation: the Arabic part of CoNLL-2012 has roughly 300k tokens, about a third the size of the English and Chinese portions.
  • Design: the game follows Wormingo's motivation-annotation paradigm, with a separate puzzle stage before the annotation stage.
    • The puzzle stage is a torn-scroll cloze worth 10 points per correct word.
    • Players find chests using a red/yellow/green distance indicator.
  • Annotation questions follow the Phrase Detectives format. For each highlighted mention, the player marks it as discourse-new, marks it as discourse-old and picks the nearest antecedent from suggested candidates, or skips it. A separate validation mode checks answers from the model or from other players.
  • Pipeline:
    • Pre-annotation uses the first neural Arabic coreference resolver (Aloraini et al., 2020), which outperformed Björkelund and Kuhn (2014).
    • Aggregation uses Mention Pair Annotation (Paun et al., 2018).
  • Stated limitations:
    • Rewards for annotation arrive only later, which could discourage players.
    • Players cannot choose an antecedent that the system did not suggest.
    • No evaluation of collected annotations is reported.

Interesting ideas

  • I1. Running an automatic resolver first lets human effort focus on checking and correcting its links, ideally where the model is uncertain, much like active learning. This makes each human judgment more valuable. — For Mova (learn): Silver coreference from our pipeline could be routed to an LLM teacher or a human reviewer only where the witness parsers or resolvers disagree, which saves labeling budget.
  • I2. Each mention gets a three-way choice: new entity, old entity with the nearest antecedent, or skip. This keeps the decision simple and avoids forcing guesses. — For Mova (frames): Our coreference step could return an explicit "new entity" or "abstain" decision with a recorded reason, rather than always linking, which keeps the decision explainable.
  • I3. Game mechanics that motivate players are kept apart from the labeling step, so that entertainment elements do not distort the labels. — For Mova (learn): When we design self-play or LLM-teacher prompts, the step that generates tasks should stay separate from the step that elicits labels, so that artifacts of the first do not leak into the training data.
  • I4. A validation mode lets contributors judge answers given by the model or by peers. This yields agreement signals in addition to fresh annotations. — For Mova (learn): Coreference links from one witness could be presented to the others as yes/no checks, which gives cheap agreement scores for filtering silver data.
  • I5. A cloze-style gap-filling puzzle placed before annotation makes readers engage with the text, so they understand it before they label references. — For Mova (answer): Our readers could fill masked words in a fable before answering coreference-dependent questions, which works as a comprehension sanity check and a negative control.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.37

@inproceedings{aliady2022coreference,
  title = {{Coreference Annotation of an Arabic Corpus using a Virtual World Game}},
  author = {Aliady, Wateen Abdullah and Aloraini, Abdulrahman and Madge, Christopher and Yu, Juntao and Bartle, Richard and Poesio, Massimo},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.37},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.37}
}

References

42 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdulrahman Aloraini, Juntao Yu, and Massimo Poesio. 2020. Neural coreference resolution for Arabic. In Proceedings of the Third Workshop on Computational Models of Reference, Anaphora and Coreference. find our card
  2. Fatima Althani, Chris Madge, and Massimo Poesio. 2022. Less text, more visuals: Evaluating the on-boarding phase in a GWAP for NLP. In Proceedings of the 9th Workshop on Games and Natural Language Processing within the 13th Language Resources and Evaluation Conference, pages 17–27, Marseille, France. European Language Resources Association. find
  3. Anders Björkelund and Jonas Kuhn. 2014. Learning structured perceptrons for coreference resolution with latent antecedents and non-local features. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 47–57, Baltimore, Maryland. Association for Computational Linguistics. find our card
  4. Federico Bonetti and Sara Tonelli. 2020. A 3D role-playing game for abusive language annotation. In Workshop on Games and Natural Language Processing, pages 39–43. find
  5. Federico Bonetti and Sara Tonelli. 2021. Challenges in designing games with a purpose for abusive language annotation. In Proceedings of the First Workshop on Bridging Human–Computer Interaction and Natural Language Processing, pages 60–65. find
  6. Saoussen Mathlouthi Bouzid and Chiraz Ben Othmane Zribi. 2020. A generic approach for pronominal anaphora and zero anaphora resolution in arabic language. Procedia Computer Science, 176:642–652. find
  7. Jon Chamberlain, Massimo Poesio, and Udo Kruschwitz. 2008. Phrase detectives: A web-based collaborative annotation game. In in Proceedings of the International Conference on Semantic Systems (I-Semantics 08), pages 42–49. find
  8. Christopher Cieri, James Fiumara, and Jonathan Wright. 2021. Using games to augment corpora for language recognition and confusability. In Proc. of Interspeech: 22nd Annual Conference of the International Speech Communication. find
  9. Dagmara Dziedzic. 2016. Use of the free to play model in games with a purpose: the robocorp game case study. Bio-Algorithms and Med-Systems, 12(4):187– 197. find
  10. Karën Fort, Bruno Guillaume, and Hadrien Chastant. 2014. Creating zombilingo, a game with a purpose for dependency syntax annotation. In Proceedings of the First International Workshop on Gamification for Information Retrieval, pages 2–6. find
  11. David Jurgens and Roberto Navigli. 2014. It’s all fun and games until someone annotates: Video games with a purpose for linguistic annotation. Transactions of the Association for Computational Linguistics, 2:449–464. find
  12. Lin Kassem, Caroline Sabty, Nada Sharaf, Menna Bakry, and Slim Abdennadher. 2016. tashkeelwap: A game with a purpose for digitizing arabic diacritics. find
  13. Alexander Kawrykow, Gary Roumanis, Alfred Kam, Daniel Kwak, Clarence Leung, Chu Wu, Eleyine Zarour, Luis Sarmenta, Mathieu Blanchette, and Jérôme Waldispühl. 2012. Phylo: A citizen science approach for improving multiple sequence alignment. PloS one, 7:e31362. find
  14. Doruk Kicikoglu, Richard Bartle, Jon Chamberlain, , and Massimo Poesio. 2019. Wormingo: a’true gamification’approach to anaphoric annotation. In Proceedings of the 14th International Conference on the Foundations of Digital Games, pages 1–7. find
  15. Robert Kleffner, Jeff Flatten, Andrew Leaver-Fay, David Baker, Justin B Siegel, Firas Khatib, and Seth Cooper. 2017. Foldit standalone: a video game-derived protein structure manipulation interface using rosetta. Bioinformatics, 33(17):2765–2767. find
  16. Markus Krause, Aneta Takhtamysheva, Marion Witt-stock, and Rainer Malaka. 2010. Frontiers of a paradigm: exploring human computation with digital games. In Proceedings of the ACM SIGKDD workshop on human computation, pages 22–25. find
  17. Mathieu Lafourcade. 2007. Making people play for lexical acquisition with the jeuxdemots prototype. In SNLP’07: 7th international symposium on natural language processing, page 7. find
  18. Mathieu Lafourcade, Alain Joubert, and Nathalie Le Brun. 2015. Games with a Purpose (GWAPs). Wiley. find
  19. Belinda Z. Li, Gabriel Stanovsky, and Luke Zettlemoyer. 2020. Active learning for coreference resolution using discrete annotation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8320–8331, Online. Association for Computational Linguistics. find our card
  20. Chris Madge, Richard Bartle, Jon Chamberlain, Udo Kruschwitz, and Massimo Poesio. 2019a. The design of a clicker game for text labelling. In 2019 IEEE Conference on Games (CoG), pages 1–4. IEEE. find
  21. Chris Madge, Richard Bartle, Jon Chamberlain, Udo Kruschwitz, and Massimo Poesio. 2019b. Incremental game mechanics applied to text annotation. In in Proceedings of the Annual Symposium on Computer-Human Interaction in Play, pages 545–558. find
  22. Sara Nasser, Nada Sharaf, Mohamed Khamis, Slim Abdennadher, and Caroline Sabty. 2013. Collecting arabic dialect variations using games with a purpose: A case study targeting the egyptian dialect. In Proceedings of the 2nd Workshop on Games and Natural Language Processing (GAMNLP 2013). find
  23. Maya Osman, Caroline Sabty, Nada Sharaf, and Slim Abdennadher. 2015. Building a corpus for arabic dialects using games with a purpose. In in 2015 First International Conference on Arabic Computational Linguistics (ACLing), IEEE, pages 21–25. find
  24. Silviu Paun, Jon Chamberlain, Udo Kruschwitz, Juntao Yu, and Massimo Poesio. 2018. A probabilistic annotation model for crowdsourcing coreference. http://aclweb.org/anthology/D18-1000, pages 1926– 1937. ↗ our card
  25. Massimo Poesio, Jon Chamberlain, and Udo Kruschwitz. 2017. Phrase Detectives, pages 1149– 1176. find
  26. Massimo Poesio, Jon Chamberlain, Udo Kruschwitz, Livio Robaldo, and Luca Ducceschi. 2013a. Phrase detectives: Utilizing collective intelligence for internet-scale language resource creation. ACM Transactions on Interactive Intelligent Systems (TiiS), 3(1):1–44. find
  27. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 254–263. find our card
  28. Kathleen Tuite. 2014. Gwaps: Games with a problem. In FDG. find
  29. Noortje J. Venhuizen, Valerio Basile, Kilian Evang, and Johan Bos. 2013. Gamification for word sense labeling. In Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Short Papers, pages 397–403, Potsdam, Germany. Association for Computational Linguistics. find
  30. Luis Von Ahn. 2006. Games with a purpose. Computer, 39(6):92–94. find
  31. Luis Von Ahn and Laura Dabbish. 2005. Esp: Labeling images with a computer game. In AAAI spring symposium: Knowledge collection from volunteer contributors, volume 2. find
  32. Massimo Poesio, Jon Chamberlain, Udo Kruschwitz, Livio Robaldo, and Luca Ducceschi. 2013b. Phrase detectives: Utilizing collective intelligence for internet-scale language resource creation. ACM Transactions on Interactive Intelligent Systems (TiiS), 3(1):1–44. find
  33. Luis Von Ahn, Mihir Kedia, and Manuel Blum. 2006a. Verbosity: a game for collecting common-sense facts. In Proceedings of the SIGCHI conference on Human Factors in computing systems, pages 75–78. find
  34. Massimo Poesio, Jon Chamberlain, Silviu Paun, Juntao Yu, Alexandra Uma, and Udo Kruschwitz. 2019. A crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), volume 1, pages 1778– 1789. Association for Computational Linguistics. find our card
  35. Luis Von Ahn, Ruoran Liu, and Manuel Blum. 2006b. Peekaboom: a game for locating objects in images. In Proceedings of the SIGCHI conference on Human Factors in computing systems, pages 55–64. find
  36. Massimo Poesio, Udo Kruschwitz, and Jon Chamberlain. 2008. ANAWIKI: Creating anaphorically annotated resources through web cooperation. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Association (ELRA). find
  37. Mingzhu Wu, Nafise Sadat Moosavi, Dan Roth, and Iryna Gurevych. 2021. Coreference reasoning in machine reading comprehension. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5768–5781, Online. Association for Computational Linguistics. find our card
  38. Massimo Poesio, Roland Stuckardt, and Yannick Versley. 2016. Anaphora resolution. Springer. find
  39. Jérôme Waldispühl, Attila Szantner, Rob Knight, Sébastien Caisse, and Randy Pitchford. 2020. Leveling up citizen science. Nature Biotechnology, 38:1124–1126. find
  40. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. pages 1–40. find our card
  41. Caroline Sabty, Mirna Yacout, Mohamed Sameh, and Slim Abdennadher. 2016. Gamified collection of arabic named entity recognition data. find
  42. Nitin Seemakurty, Jonathan Chu, Luis von Ahn, and Anthony Tomasic. 2010. Word sense disambiguation via human computation. In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP ’10, page 60–63, New York, NY, USA. Association for Computing Machinery. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aliady-2022-coreference-annotation-arabic-corpus-virtual/