Elamur Research⌕
☰
home / papers / 2014 / 3arif: A Corpus of Modern Standard and Egyptian Arabic Tweets Annotated for Epistemic Modality Using Interactive Crowdsourcing

3arif: A Corpus of Modern Standard and Egyptian Arabic Tweets Annotated for Epistemic Modality Using Interactive Crowdsourcing

Paper · 2014 · Rania Al-Sabbagh, Roxana Girju and Jana Diesner

Summary

Arabic had no large corpus marked for epistemic modality, meaning the linguistic signals of how certain a speaker is about a proposition. Existing modality schemes usually need annotators with linguistics training, which keeps such projects small and confined to the lab. The authors built 3arif, a collection of 9822 tweets in Modern Standard Arabic and Egyptian Arabic. The tweets come from newspapers, TV stations, political campaigns and private users.

Candidate triggers were first flagged by string matching against existing lexicons. Crowd annotators then answered an adaptive questionnaire. A synonymy check against a set of clear example triggers decides whether a candidate really expresses epistemic meaning. Only if it does do further questions appear on polarity, intensification, tense, holder and scope. Nested holders, such as quoted or attributed beliefs, are collected through simple questions, and the system merges triggers that share holders and scopes.

Reliability, measured with Krippendorff's alpha and raw agreement, was high and comparable to expert-annotated schemes. The paper therefore gives both a new resource and evidence that complex semantic annotation can be split into questions non-experts can answer.

Key points

  • The corpus has 9822 unique tweets with 9966 candidate trigger tokens, covering 214 unique epistemic modality types, in MSA and EA.
  • Each trigger is labelled in six tasks: sense, polarity, intensification (AMP / MTG / AS IS), tense (PRS / PST only), holder(s) and scope(s).
  • By majority vote, 5591 candidates are epistemic and 4375 are not. Among the epistemic ones:
    • polarity: 3425 TRUE, 2166 FALSE;
    • intensification: 1083 amplified, 330 mitigated, 4178 as is;
    • tense: 4399 present, 1192 past.
  • About 62% of triggers have no nested holder, so the tweet author is the holder. Of the scopes, clauses make up 86% and deverbal nouns 9%, with to-infinitives accounting for the remainder.
  • Tense annotation reached near-perfect reliability. The main confusion came from inchoative phrases like "I started to believe".
  • Future tense, interrogative, imperative and hypothetical moods are not covered, and neither is modality entailment (e.g. "whoever thinks X is wrong").

Interesting ideas

  • I1. Whether a word is epistemic is decided by asking if it means the same as a curated set of unambiguous exemplars, and that set is chosen to match the candidate's inflection, so lay annotators never need a definition of modality.
  • I2. Annotators who answer "no" must also type their own synonym, which stops them using the negative option as a shortcut.
  • I3. The questionnaire shows follow-up questions only when earlier answers make them relevant, and the authors openly note that this spreads early disagreements into later tasks.
  • I4. The holder is kept separate from the grammatical subject, and holders can be nested through quotation or belief attribution, for example a user reporting what a government seems to think.
  • I5. Annotators must give a reason for every polarity, intensity and tense choice, naming the cue: a negation particle, double negation, an emphatic word, coordination of triggers, and so on.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): An LLM teacher tier could judge "is this verb used in sense X?" by comparing it with a few clear-cut exemplar sentences instead of an abstract gloss, which makes silver sense labels easier to check.
  • I2 (learn): When LLM teachers or witness parsers reject a candidate label, they could be required to propose an alternative, giving a cheap way to catch lazy or degenerate answers.
  • I3 (learn): Gated evaluation pipelines should record where a cascade first diverges, so that errors in later stages (e.g. frames after a wrong tree) are not blamed on the stage where they show up.
  • I4 (frames): Speech-act and belief frames could carry a stack of nested holders (author → quoted source → attributed believer) instead of defaulting to the nsubj.
  • I5 (world): Belief states in the text world could store certainty plus the cue that amplified or mitigated it (e.g. "absolutely", "almost", stacked negation), keeping the certainty value explainable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois Urbana-ChampaignUrbana · United States184 works · 69,372 citations

Also on the byline: National Center for Supercomputing Applications · National Center for Supercomputing Applications.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksLiterature and Literary Theory · 519 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Rania Al-Sabbagh, Roxana Girju and Jana Diesner. 2014. 3arif: A Corpus of Modern Standard and Egyptian Arabic Tweets Annotated for Epistemic Modality Using Interactive Crowdsourcing. Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alsabbagh20143arif,
  title = {{3arif: A Corpus of Modern Standard and Egyptian Arabic Tweets Annotated for Epistemic Modality Using Interactive Crowdsourcing}},
  author = {Rania Al-Sabbagh, Roxana Girju and Jana Diesner},
  booktitle = {Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers},
  year = {2014},
  doi = {10.13140/2.1.3402.0486},
  url = {https://doi.org/10.13140/2.1.3402.0486}
}

References

18 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed and Mona Diab. 2011. Subjectivity and Sentiment Annotation of Modern Standard Arabic Newswire. In Proceedings of the 5 th Linguistic Annotation Workshop (LAW V), pages 110-118, Portland, Oregon, June 23-24, 2011.
  2. Rania Al-Sabbagh, Jana Diesner and Roxana Girju. 2013. Using the Semantic-Syntactic Interface for Reliable Arabic Modality Annotation. In Proceedings of IJCNLP'13, pages 410-418, Nagoya, Japan, October 14-18, 2013.
  3. Rania Al-Sabbagh, Roxana Girju and Jana Diesner. 2014. Unsupervised Construction of a Lexicon and a Pattern Repository of Arabic Modal Multiword Expressions. In Proceedings of the 10 th Workshop of Multiword Expressions at EACL 2014, pages 114-123, Gothenburg, Sweden, April 26-27, 2014. find
  4. Ron Artstein and Massimo Poesio. 2008. Inter-Coder Agreement for Computational Linguistics. Computational Linguistics, volume 34, issue 4, pages 555-596. ↗
  5. Kathrin Baker, Michael Bloodgood, Mona Diab, Bonnie Dorr, Nathaniel W. Filardo, Lori Levin and Christine Piatko. 2010. A Modality Lexicon and its Use in Automatic Tagging. In Proceedings of LREC'10, pages 1402-1407, Valetta, Malta, May 19-21, 2010. find
  6. Kathryn Baker, Michael Bloodgood, Bonnie J. Dorr, Chris Callison-Burch, Nathaniel W. Filardo, Christine Piatko, Lori Levin and Scott Miller. 2012. Modality and Negation in SIMT. Computational Linguistics. volume 38, issue 2, pages 411-438. find
  7. Farah Benamara, Baptiste Chardon, Yannick Mathieu, Vladimir Popescu and Nicholas Asher. 2012. How do Negation and Modality Impact on Opinions. In Proceedings of the ACL-2012 Workshop on ExProM-2012, pages 10-18, Jeju, Republic of Korea, July 13, 2012. find
  8. Yanyan Cui and Ting Chi. 2013. Annotating Modal Expressions in the Chinese Treebank. In Proceedings of the IWC 2013 Workshop on Annotation of Modal Meaning in Natural Language (WAMM), pages 24-32, Potsdam, Germany, March 19, 2013. find
  9. Mona Diab, Lori Levin, Teruko Mitamura, Owen Rambow, Vinodkumar Prabhakaran, and Weiwei Guo. 2009. Committed Belief Annotation and Tagging. In Proceedings of the 3 rd Linguistic Annotation Workshop, ACL-IJCNLP'09, pages 68-73, Suntec, Singapore, August 6-7, 2009.
  10. Iris Hendrickx, Amàlia Mendes and Silvia Mencarelti. 2012. Modality in Text: A Proposal for Corpus Annotation. In Proceedings of the 8 th International Conference on Language Resources and Evaluation (LREC'12), pages 1805-1812, Istanbul, Turkey, May 21-27, 2012. find
  11. Klaus Krippendorff. 2011. Computing Krippendorff's Alpha Reliability. Annenberg School of Communication, Departmental Papers: University of Pennsylvania. find
  12. Suguru Matsuyoshi, Megumi Eguchi, Chitose Sao, Koji Murakami, Kentaro Inui and Yuji Matsumoto. 2010. Annotating Event Mentions in Text with Modality Focus and Source Information. In Proceedings of LREC'10, pages 1456-1463, Valletta, Malta, May 19-21, 2010. find
  13. Frank R. Palmer. 2001. Mood and Modality. 2 nd Edition. Cambridge University Press, Cambridge, UK. ↗
  14. Aynat Rubinstein, Hillary Harner, Elizabeth Krawczyk, Daniel Simoson, Graham Katz and Paul Portner. 2013. Toward Fine-Grained Annotation of Modality in Text. In Proceedings of the IWC 2013 Workshop on Annotation of Modal Meaning in Natural Language (WAMM), pages 38-46, Potsdam, Germany, March 19, 2013. find
  15. Roser Saurí and James Pustejovsky. 2009. FactBank: A Corpus Annotated with Event Factuality. Language Resources and Evaluation, volume 43, pages 227-268. ↗
  16. György Szarvas, Veronika Vincze, Richárd Farkas and János Csirik. 2008. The BioScope Corpus: Annotation for Negation, Uncertainty and their Scope in Biomedical Texts. In Proceedings of BioNLP 2008: Current Trends in Biomedical Natural Language Processing, pages 38-45, Columbus, Ohio, USA, June 2008.
  17. Anita de Waard and Henk Pander Maat. 2012. Epistemic Modality and Knowledge Attribution in Scientific Discourse: a Taxonomy of Types and Overview of Features. In Proceedings of the 50 th ACL, pages 47-55, Jeju, Republic of Korea, July 12, 2012.
  18. Janyce Wiebe, Theresa Wilson and Claire Cardie. 2005. Annotating Expressions of Opinions and Emotions in Language. Language Resources and Evaluation, volume 39, issue 2-3, pages 163-210. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sabbagh-2014-3arif-corpus-modern-standard-egyptian/