Elamur Research⌕
☰
home / papers / 2013 / Supersense Tagging for Arabic: the MT-in-the-Middle Attack

Supersense Tagging for Arabic: the MT-in-the-Middle Attack

Paper · 2013 · Nathan Schneider, Behrang Mohit, Chris Dyer et al.

Summary

The paper tackles nominal supersense tagging for Arabic: finding noun chunks and assigning each one of the 25 coarse WordNet noun categories. Arabic has too little annotated text for a supervised tagger, so the authors keep the small AQMAR Arabic Wikipedia Supersense Corpus for development and testing only. They compare three approaches that need no Arabic training labels:

  • a lexicon built from Arabic WordNet and OntoNotes, used with greedy matching and sense-order heuristics;
  • unsupervised feature-based sequence clustering on Arabic Wikipedia text;
  • "MT-in-the-middle": translate the Arabic sentence to English, tag the English output with an existing supersense tagger, then carry the labels back to Arabic through the alignments.

MT-in-the-middle works best. The cdec system wins even though QCRI translates better, because cdec exposes word-level alignments. Adding lexicon lookups to the MT approach raises recall further. Results remain well below the agreement between human annotators. The work matters as the first study of Arabic supersense tagging and as evidence that imperfect MT can still carry lexical semantics across languages.

Key points

  • Test data: 18 Wikipedia articles (1,393 sentences, 39,916 tokens) from the history, science, sports and technology domains. Each of the two annotators, Ann-A and Ann-B, is scored as a separate test set.
  • Lexicon baseline:
    • The lexicon has 23,000 types, 11,000 of them multiword.
    • It covers only 18% of test tokens, and in-vocabulary tokens average 2.0 possible supersenses.
    • Labeled F1 is only about 20 (21.6 against Ann-A, 19.4 against Ann-B).
  • MT-in-the-middle:
    • With cdec, labeled F1 is 33.8 / 34.6; with QCRI it is 29.9 / 31.6.
    • The hybrid of cdec plus lexicon reaches 35.5 / 36.0, with noun coverage rising to 68% (the gold standard covers 77% of nouns).
    • Inter-annotator F1 is about 70.
  • MT quality versus tagging usefulness:
    • QCRI scores higher on standard metrics (BLEU 32.86 vs 28.84).
    • It also has higher lexical projection precision (91.9% vs 90.0%).
    • Yet cdec is more useful for tagging, because QCRI's phrase-level alignments blur mention boundaries.
  • Error sources:
    • Domain mismatch produces wrong lexical choices, e.g. "corn" for atom.
    • Transliteration fails on names, e.g. "IMAX" for EMACS.
    • Overall, translation errors explain only part of the gap.

Interesting ideas

  • I1. Instead of projecting labels across a parallel corpus and training a new tagger, translate each test sentence at run time, tag the translation with a strong existing tool, and map the labels back; this avoids needing in-domain parallel data.
  • I2. Alignment granularity can matter more than translation quality: a weaker translator with word-level links beat a better one with only phrase-level links for label transfer.
  • I3. Projection is restricted by a small syntactic rule: only nouns, plus adjectives in a chain after a noun, receive tags, and all target words linked to one source mention are merged into a single span.
  • I4. Lexicon ambiguity is resolved by weighting each candidate category by how many of the word's senses map to it, discounted by the rank of its earliest sense.
  • I5. Lexical projection precision, where a bilingual judge checks only the tagged mentions in the MT output, isolates translation damage from tagging damage better than BLEU does.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Translation through the meta-world could reuse this route in reverse, sending foreign input into English so the full Mova pipeline can analyze it, then projecting frames back along the alignment.
  • I2 (learn): When choosing teachers or witness systems for silver data, rank them by how precisely their outputs can be anchored to source tokens, not only by their headline scores.
  • I3 (frames): Similar explicit rules can turn aligned or projected labels into clean argument spans, keeping each mapping step explainable.
  • I4 (world): This gives a cheap, transparent default supersense for WordNet-derived ontology lookups when no context model is available.
  • I5 (learn): Evaluations could add targeted human or LLM checks on just the spans a stage depends on, which shows where errors originate.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 1,087 worksSemitic languages · 2,908 works

Cite

Cite as Nathan Schneider, Behrang Mohit, Chris Dyer, Kemal Oflazer and Noah A. Smith. 2013. Supersense Tagging for Arabic: the MT-in-the-Middle Attack. Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{schneider2013supersense,
  title = {{Supersense Tagging for Arabic: the MT-in-the-Middle Attack}},
  author = {Nathan Schneider, Behrang Mohit, Chris Dyer, Kemal Oflazer and Noah A. Smith},
  booktitle = {Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2013},
  url = {https://aclanthology.org/N13-1076/}
}

References

35 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan, June. Association for Computational Linguistics.
  2. Taylor Berg-Kirkpatrick, Alexandre Bouchard-Côté, John DeNero, and Dan Klein. 2010. Painless unsupervised learning with features. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT 2010), pages 582–590, Los Angeles, California, June. Association for Computational Linguistics.
  3. David Burkett and Dan Klein. 2008. Two languages are better than one (for syntactic parsing). In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP 2008), pages 877–886, Honolulu, Hawaii, October. Association for Computational Linguistics. find
  4. David Burkett, Slav Petrov, John Blitzer, and Dan Klein. 2010. Learning better monolingual models with unannotated bilingual text. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL 2010), pages 46–54, Uppsala, Sweden, July. Association for Computational Linguistics. find
  5. Stanley F. Chen and Joshua Goodman. 1999. An empirical study of smoothing techniques for language modeling. Computer Speech & Language, 13(4):359–393, October. ↗
  6. David Chiang. 2007. Hierarchical phrase-based translation. Computational Linguistics, 33(2):201–228, June. ↗
  7. Kenneth W. Church and Eduard H. Hovy. 1993. Good applications for crummy machine translation. Machine Translation, 8(4):239–258, December. ↗
  8. Massimiliano Ciaramita and Yasemin Altun. 2006. Broad-coverage sense disambiguation and information extraction with a supersense sequence tagger. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 594–602, Sydney, Australia, July. Association for Computational Linguistics.
  9. Massimiliano Ciaramita and Mark Johnson. 2003. Supersense tagging of unknown nouns in WordNet. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, pages 168– 175, Sapporo, Japan, July.
  10. Dipanjan Das and Slav Petrov. 2011. Unsupervised part-of-speech tagging with bilingual graph-based projections. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT 2011), pages 600–609, Portland, Oregon, USA, June. Association for Computational Linguistics.
  11. Chris Dyer, Adam Lopez, Juri Ganitkevitch, Jonathan Weese, Ferhan Ture, Phil Blunsom, Hendra Setiawan, Vladimir Eidelman, and Philip Resnik. 2010. cdec: A decoder, alignment, and learning framework for finite-state and context-free translation models. In Proceedings of the ACL 2010 System Demonstrations, pages 7–12, Uppsala, Sweden, July. Association for Computational Linguistics.
  12. Sabri Elkateb, William Black, Horacio Rodríguez, Musa Alkhalifa, Piek Vossen, Adam Pease, and Christiane Fellbaum. 2006. Building a WordNet for Arabic. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC 2006), pages 29–34, Genoa, Italy, May.
  13. Christiane Fellbaum, editor. 1998. WordNet: an electronic lexical database. MIT Press, Cambridge, MA. find
  14. Michael Gamon, Anthony Aue, and Martine Smets. 2005. Sentence-level MT evaluation without reference translations: Beyond language modeling. In Proceedings of the 10th European Association for Machine Translation Conference (EAMT 2005), pages 103–111, Budapest, May. find
  15. Nizar Habash and Owen Rambow. 2005. Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 573–580, Ann Arbor, Michigan, June. Association for Computational Linguistics.
  16. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: the 90% solution. In Proceedings of the Human Language Technology Conference of the NAACL (HLT-NAACL 2006), pages 57–60, New York City, USA, June. Association for Computational Linguistics. find
  17. Rebecca Hwa, Philip Resnik, Amy Weinberg, Clara Cabezas, and Okan Kolak. 2005. Bootstrapping parsers via syntactic projection across parallel texts. Natural Language Engineering, 11(3):311–325, September. ↗
  18. Sungchul Kim, Kristina Toutanova, and Hwanjo Yu. 2012. Multilingual named entity recognition using parallel data and metadata from Wikipedia. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL 2012), pages 694–702, Jeju Island, Korea, July. Association for Computational Linguistics. find
  19. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology (HLT-NAACL 2003), pages 48–54, Edmonton, Canada. Association for Computational Linguistics. ↗
  20. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Demo and Poster Sessions, pages 177–180, Prague, Czech Republic, June. Association for Computational Linguistics. find
  21. Alon Lavie and Michael J. Denkowski. 2009. The METEOR metric for automatic evaluation of machine translation. Machine Translation, 23(2–3):105–115, September. ↗
  22. Mohamed Maamouri, Ann Bies, Tim Buckwalter, and Wigdan Mekki. 2004. The Penn Arabic Treebank: building a large-scale annotated Arabic corpus. In NEMLAR Conference on Arabic Language Resources and Tools, pages 102–109. find
  23. Bernard Merialdo. 1994. Tagging English text with a probabilistic model. Computational Linguistics, 20(2):155–171.
  24. Rada Mihalcea, Carmen Banea, and Janyce Wiebe. 2007. Learning multilingual subjective language via cross-lingual projections. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (ACL 2007), pages 976–983, Prague, Czech Republic, June. Association for Computational Linguistics.
  25. George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. 1993. A semantic concordance. In Proceedings of the Workshop on Human Language Technology (HLT ’93), pages 303–308, Plainsboro, NJ, USA, March. Association for Computational Linguistics.
  26. Behrang Mohit, Nathan Schneider, Rishav Bhowmick, Kemal Oflazer, and Noah A. Smith. 2012. Recall-oriented learning of named entities in Arabic Wikipedia. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2012), pages 162–173, Avignon, France, April. Association for Computational Linguistics.
  27. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics (ACL 2002), pages 311–318, Philadelphia, Pennsylvania, USA, July. Association for Computational Linguistics.
  28. Altaf Rahman and Vincent Ng. 2012. Translation-based projection for multilingual coreference resolution. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2012), pages 720–730, Montréal, Canada, June. Association for Computational Linguistics.
  29. Ryan Roth, Owen Rambow, Nizar Habash, Mona Diab, and Cynthia Rudin. 2008. Arabic morphological tagging, diacritization, and lemmatization using lexeme models and feature ranking. In Proceedings of ACL-08: HLT, pages 117–120, Columbus, Ohio, June. Association for Computational Linguistics.
  30. Nathan Schneider, Behrang Mohit, Kemal Oflazer, and Noah A. Smith. 2012. Coarse lexical semantic annotation with supersenses: an Arabic case study. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL 2012), pages 253–258, Jeju Island, Korea, July. Association for Computational Linguistics.
  31. David A. Smith and Noah A. Smith. 2004. Bilingual parsing with factored estimation: using English to parse Korean. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP’04), pages 49–56, Barcelona, Spain. find
  32. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (AMTA 2006), pages 223–231, Cambridge, Massachusetts, USA, August.
  33. David Yarowsky and Grace Ngai. 2001. Inducing multilingual POS taggers and NP bracketers via robust projection across aligned corpora. In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL’01), Pittsburgh, Pennsylvania, USA, June. find
  34. David Yarowsky, Grace Ngai, and Richard Wicentowski. 2001. Inducing multilingual text analysis tools via robust projection across aligned corpora. In Proceedings of the First International Conference on Human Language Technology Research (HLT’01), San Diego, California, USA, March.
  35. Imed Zitouni and Radu Florian. 2008. Mention detection crossing the language barrier. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP 2008), pages 600–609, Honolulu, Hawaii, October. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schneider-2013-supersense-tagging-arabic-mt-middle/