Elamur Research⌕
☰
home / papers / 2014 / Can Crowdsourcing be used for Effective Annotation of Arabic?

Can Crowdsourcing be used for Effective Annotation of Arabic?

Paper · 2014 · Wajdi Zaghouani and Kais Dukes

Summary

The paper asks whether paid non-expert workers on Amazon Mechanical Turk (AMT) can do reliable linguistic annotation of Arabic. Expert annotation is slow and costly, and crowdsourcing had worked for several English NLP tasks. The authors drew a small test set from the opening of chapter 23 of the Quranic Arabic Corpus, a section an expert linguist had re-verified, and used its morphological annotation as gold. They ran two multiple-choice tasks behind an Arabic screening test. One asked workers to tag words with a coarse part-of-speech set. The other asked them to pick the grammatical case ending of each word. Workers earned one cent per word. Few people qualified or stayed with the task. POS accuracy reached about 64% and case accuracy about 50%, which is above the chance level for case but well below published automatic taggers. The authors conclude that annotation needing real grammatical expertise does not transfer easily to the crowd, and that case marking is the harder of the two tasks. The study is an early, cautionary data point for crowdsourcing annotation in morphologically rich languages.

Key points

  • Gold data came from the Quranic Arabic Corpus (Leeds), which has a morphological layer and a syntactic Treebank. The case task used the first 100 words of chapter 23, and the POS task used the first 200.
  • The case task had four options: nominative, genitive, accusative, or none. The POS task used a coarse tagset of noun, verb, adjective, pronoun, particle, and other/unknown. The text calls this set 5 tags in one place and lists six choices in another.
  • Of 137 interested Turkers, only 24 passed the qualification test, and 17 actually annotated. Workers who did only a handful of HITs were excluded, and only 4 completed every HIT assigned to them.
  • Case tagging used 7 annotators, completed 657 of 700 HITs, and reached 50.07% accuracy against a random baseline of about 25%. POS tagging used 10 annotators, completed 690 of 2000 HITs, and reached 63.91% accuracy, with individual annotators ranging from roughly 42% to 80%.
  • The authors blame low participation on three things: a one-month window, pay of $0.01 per word, and the difficulty of the task. They note that automatic Arabic taggers report roughly 80–90% accuracy.

Interesting ideas

  • I1. The authors check whether workers understood the task by comparing their accuracy to the random-choice rate for the label set (about 25% with four options). This separates confusion about the instructions from a lack of skill.
  • I2. The test set was deliberately restricted to a slice of the gold corpus that an expert had re-verified, instead of the whole, still-changing corpus.
  • I3. Turning a complex annotation into a short closed choice per token makes the work checkable. It also shows which subtask is hard: case required syntax, while POS needed only word-level knowledge.
  • I4. The paper finds that labels decided by syntactic function are much harder for non-experts than surface labels. Its example is Arabic sound masculine plurals, where accusative and genitive share one ending.
  • I5. Per-annotator accuracy varied widely, and workers who did very few items were dropped. Volume and reliability were weighed together rather than pooled.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Every silver-data witness or LLM teacher tier could be scored against a uniform-random and a majority-label control. A source that barely beats these controls would be treated as noise rather than signal.
  • I2 (learn): Evaluation of the parser and frames should be anchored on a small, doubly-checked subset of each treebank. That keeps measured error separate from annotation error in the gold.
  • I3 (tokens): When LLM teachers are asked for morphology labels, framing each query as a small closed set per feature would let errors be traced to the feature level, for example case versus POS.
  • I4 (tree): Agreement among witness parsers could be tracked separately for surface-ambiguous forms. Such forms should then route to a stronger teacher or a dedicated repair rule rather than a cheap majority vote.
  • I5 (learn): Each witness should get its own accuracy on a gold probe, and votes should be weighted by that accuracy. Low-coverage witnesses should not be allowed to tip the agreement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓University of LeedsLeeds · United Kingdom40 works · 1,822 citations🎓Carnegie Mellon University QatarDoha · Qatar47 works · 2,773 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Science Applications · 102 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Wajdi Zaghouani and Kais Dukes. 2014. Can Crowdsourcing be used for Effective Annotation of Arabic?. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zaghouani2014can,
  title = {{Can Crowdsourcing be used for Effective Annotation of Arabic?}},
  author = {Wajdi Zaghouani and Kais Dukes},
  booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)},
  year = {2014},
  doi = {10.63317/5a4f4su6miyg},
  url = {https://doi.org/10.63317/5a4f4su6miyg}
}

References

20 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adda, G., Sagot, B., Fort, K., Mariani, J. (2011). Crowdsourcing for Language Resource Development: Critical Analysis of Amazon Mechanical Turk Overpowering Use. In Proceedings of the Language and Technology Conference, Poznan, Poland. find
  2. Alkuhlani, S., Habash, N. & Roth, R. (2013). Automatic Morphological Enrichment of a Morphologically Underspecified Treebank. In Proceedings of Conference of the North American Association for Computational Linguistics (NAACL), Atlanta, Georgia. find
  3. Bloodgood, M., & Callison-Burch, C. (2010). Using mechanical turk to build machine translation evaluation sets. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon’s Mechanical Turk. Los Angeles, Ca.
  4. Callison-Burch, C., & Dredze, M. (2010). Creating speech and language data with amazon’s mechanical turk. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon’smechanical Turk. Los Angeles, Ca.
  5. Callison-Burch, C. (2009). Fast, cheap, and creative: Evaluating translation quality using Amazon’s Mechanical Turk. In Proceedings of the conference on empirical methods in natural language processing (EMNLP 2009). Singapore, Singapore.
  6. Chamberlain, J.; Poesio, M. & Kruschwitz. (2009). A new life for a dead parrot: Incentive structures in the Phrase Detectives game. In Proceedings of the Webcentives Workshop at WWW’09. Madrid, Spain. find
  7. Dukes, K. (2013). Statistical Parsing by Machine Learning from a Classical Arabic Treebank. PhD Thesis, University of Leeds.
  8. Dukes, K. & Buckwalter, T. (2010). A Dependency Treebank of the Quran using Traditional Arabic Grammar. In Proceedings of the Seventh International conference on Informatics and Systems. Cairo, Egypt. find
  9. Dukes, K. & Habash, N. (2010). Morphological Annotation of Quranic Arabic. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC 2010). Valletta, Malta.
  10. Dukes, K., Atwell, E. & Sharaf, A-B. (2010). Syntactic Annotation Guidelines for the Quranic Arabic Treebank. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC 2010). Valletta, Malta. find
  11. Habash, N., Gabbard, R., Rambow, O., Kulick, S. & Marcus, M. (2007). Determining Case in Arabic: Learning Complex Linguistic Behavior Requires Complex Linguistic Features. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Prague, Czech Republic.
  12. Habash, N. & Rambow, O. (2005). Arabic Tokenization, Morphological Analysis, and Part-of-Speech Tagging in One Fell Swoop. In Proceedings of the Conference of the Association for Computational Linguistics (ACL’05), Ann Arbor, MI. find
  13. Maamouri, M., Bies, A., Kulick, S. (2008). Enhanced Annotation and Parsing of the Arabic Treebank. In Proceedings of the INFOS International Conference, Cairo, Egypt. find
  14. Maamouri, M., Kulick, S. & Bies, A. (2008). Diacritic Annotation in the Arabic Treebank and Its Impact on Parser Evaluation. In Proceedings of the International Conference on Language Resources and Evaluation (LREC), Marrakech, Morocco.
  15. Ryding, K-C. (2008). A Reference Grammar of Modern Standard Arabic. Cambridge University Press. find
  16. Siddiqui, A-R. (2008). Quranic Keywords: A Reference Guide. The Islamic Foundation. find
  17. Smrž, O. & Hajič, J. (2006). The Other Arabic Treebank: Prague Dependencies and Functions. In Arabic Computational Linguistics: Current Implementations, CSLI Publications. find
  18. Snow, R., O’Connor, B., Jurafsky, D. and.Ng, A-Y. (2008). Cheap and fast – but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of EMNLP, Waikiki, Honolulu, Hawaii.
  19. Su, Q., Pavlov, D., Chow, J-H & Baker, W-C. (2007). Internet-Scale Collection of Human-Reviewed Data. In Proceedings of the 16th international conference on World Wide Web. Banff, Alberta, Canada. find
  20. Wang, A., Hoang, C.D-V., & Kan, M-Y. (2010). Perspectives on Crowdsourcing Annotations for Natural Language Processing. Language Resources and Evaluation,47:1. Figure 1: Task creation sample. Figure 2: A sample case ending question. Figure 3: A sample screening test question. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zaghouani-2014-can-crowdsourcing-be-used-effective/