Elamur Research⌕
☰
home / papers / 2016 / Farasa: A New Fast and Accurate Arabic Word Segmenter

Farasa: A New Fast and Accurate Arabic Word Segmenter

Paper · 2016 · Kareem Darwish and Hamdy Mubarak

Summary

Arabic words often fuse several clitics, such as conjunctions, prepositions, the article, the stem and pronoun suffixes. Splitting them correctly is a basic step for search, tagging, parsing and MT. Darwish and Mubarak present Farasa, a segmenter built to match the strongest tools of its time while running far faster.

For each word, considered without its sentence context, Farasa lists every split that the allowed prefix and suffix inventory and ordering rules permit. It then ranks these candidates with SVM rank using a linear kernel. Features include:

  • affix probabilities and prefix–suffix co-occurrence statistics,
  • unigram stem frequencies from a large Aljazeera corpus,
  • stem-template validity,
  • membership in several lexicons and a gazetteer.

The model is trained on the Penn Arabic Treebank (ATB). The authors argue that testing inside the ATB inflates scores, so they built a new WikiNews test set from fresh news text. On it, Farasa ties or beats MADAMIRA and QATARA while being roughly 10–100 times faster. It is pure Java and open-source, which makes large-scale Arabic processing cheap.

Key points

  • Error rates on WikiNews (tokenization error): MADAMIRA 1.24%, QATARA 1.77%, Farasa Base 1.24%, Farasa Lookup 1.06%.
  • Speed: roughly 10 times quicker than QATARA and about 100 times quicker than MADAMIRA. It can handle 1 billion words in under 5 hours, in native Java with no external dependencies.
  • Data:
    • Training uses ATB parts 1 (v4.1), 2 (v3.1) and 3 (v2).
    • The test set has 70 WikiNews articles from 2013–2014: 7 themes with 10 articles each, 18,271 words in total.
    • Stem unigram probabilities come from an Aljazeera.net corpus of 114,758 articles and 94M words, covering 2000–2011.
  • Lexical resources:
    • AraComLex (31,753 stems)
    • the Buckwalter lexicon from AraMorph
    • a Wikipedia-derived person and location gazetteer
    • a function-word list
    • a stem lexicon built by filling roots into templates
  • Learner and cache: SVM rank with a linear kernel and C=100, tuned on a dev set. The Lookup variant caches training-time segmentations only for two kinds of words: those that were never ambiguous, and those seen at least 5 times whose dominant split exceeds 70%.

Interesting ideas

  • I1. Instead of tagging characters, Farasa enumerates every split allowed by a small grammar of legal affix orders and lets a linear ranker choose among them, so each decision traces back to readable feature weights.
  • I2. The authors deliberately ignore sentence context and then show empirically that this costs very little, which buys large speed gains.
  • I3. The memorization cache is gated by confidence: a word is cached only if training showed it unambiguous, or it occurred at least 5 times with one split above 70%; everything else goes to the model.
  • I4. In-treebank test sets give inflated scores because they are lexically too close to training data, so the authors built a fresh, theme-balanced test set from a different source.
  • I5. The stem lexicon is generated combinatorially by filling roots into templates and is kept only where the forms are attested in a corpus, turning generative rules into a precise lexicon.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): English clitic and affix analysis of unknown words (e.g. -n't, 's, derivational suffixes) could be framed as ranking grammar-licensed candidates, keeping each choice explainable.
  • I2 (tokens): Mova could measure how many lemma and morphology decisions are truly context-free and route only the ambiguous remainder to the contextual tagger.
  • I3 (learn): the same frequency-plus-dominance gate could decide which silver analyses agreed by witness parsers become hard-coded rules and which stay with the trained model.
  • I4 (learn): Mova's parsers and taggers could be evaluated on new, topic-stratified texts outside the training treebanks, which exposes overfitting to treebank vocabulary.
  • I5 (world): candidate derived forms produced from WordNet/verb-class knowledge could likewise be filtered by corpus attestation before entering the compiled lexicon.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Kareem Darwish and Hamdy Mubarak. 2016. Farasa: A New Fast and Accurate Arabic Word Segmenter. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{darwish2016farasa,
  title = {{Farasa: A New Fast and Accurate Arabic Word Segmenter}},
  author = {Kareem Darwish and Hamdy Mubarak},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  doi = {10.63317/59oiyegxbo3y},
  url = {https://doi.org/10.63317/59oiyegxbo3y}
}

References

22 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aljlayl, M. and Frieder, O. (2002). On arabic search: improving the retrieval effectiveness via a light stemming approach. In CIKM-2002, pages 340–347. find
  2. Attia, M., Pecina, P., Toral, A., Tounsi, L., and van Genabith, J. (2011). An open-source finite state morphological transducer for modern standard arabic. In Proceedings of the 9th International Workshop on Finite State Methods and Natural Language Processing, pages 125– 133.
  3. Beesley, K., Buckwalter, T., and Newton, S. (1989). Two-level finite-state analysis of arabic morphology. In Proceedings of the Seminar on Bilingual Computing in Arabic and English, pages 6–7. find
  4. Beesley, K. R. (1996). Arabic finite-state morphological analysis and generation. In Proceedings of the 16th conference on Computational linguistics-Volume 1, pages 89–94. Association for Computational Linguistics.
  5. Buckwalter, T. (2002). Buckwalter {Arabic} morphological analyzer version 1.0. find
  6. Darwish, K. and Oard, D. W. (2007). Adapting morphology for arabic information retrieval*. In Arabic Computational Morphology, pages 245–262. Springer. ↗
  7. Darwish, K., Magdy, W., and Mourad, A. (2012). Language processing for arabic microblog retrieval. In ACM CIKM-2012, pages 2427–2430. find
  8. Darwish, K., Abdelali, A., and Mubarak, H. (2014). Using stem-templates to improve arabic pos and gender/number tagging. In LREC-2014.
  9. Darwish, K. (2002). Building a shallow arabic morphological analyzer in one day. In Computational Approaches to Semitic Languages, ACL-2002, pages 1–8. find
  10. Diab, M. (2009). Second generation amira tools for arabic processing: Fast and robust tokenization, pos tagging, and base phrase chunking. In 2nd International Conference on Arabic Language Resources and Tools. QATARA b+msbAr ( PAJ.‚Ó + H . ) fysbwk ( ¼ñJ.‚ ¯ ) kwrwnA ( A KðPñ» )   + ‡’ÊÓ ) mlSq+At ( H@ lynks ( º JJ Ë ) w+lmfAj>+th+mA  Ag. A ®ÖÏ+ð ) ( AÒ+î E+ Habash, N., Rambow, O., and Roth, R. (2009). Mada+ tokan: A toolkit for arabic tokenization, diacritization, morphological disambiguation, pos tagging, stemming and lemmatization. In Proceedings of the 2nd International Conference on Arabic Language Resources and Tools (MEDAR), Cairo, Egypt, pages 102–109. find
  11. Habash, N., Roth, R., Rambow, O., Eskander, R., and Tomeh, N. (2013). Morphological analysis and disambiguation for dialectal arabic. In HLT-NAACL, pages 426–432.
  12. Joachims, T. (2006). Training linear svms in linear time. In ACM SIGKDD-2006, pages 217–226. ACM. find
  13. Khoja, S. and Garside, R. (2001). Automatic tagging of an Arabic corpus using APT. Ph.D. thesis, University of Utah, Salt Lake City, Utah. find
  14. Khoja, S. (2001). Apt: Arabic part-of-speech tagger. In NAACL, pages 20–25. find
  15. Kiraz, G. A. (1998). Arabic computational morphology in the west. In Proceedings of the 6th International Conference and Exhibition on Multi-lingual Computing, pages 3–5. Citeseer. find
  16. Larkey, L. S., Ballesteros, L., and Connell, M. E. (2002). Improving stemming for arabic information retrieval: light stemming and co-occurrence analysis. In Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, pages 275–282. ACM. find
  17. Lee, Y.-S., Papineni, K., Roukos, S., Emam, O., and Hassan, H. (2003). Language model based arabic word segmentation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1, pages 399–406. Association for Computational Linguistics.
  18. Maamouri, M., Bies, A., Buckwalter, T., Jin, H., and Mekki, W. (2005). Arabic treebank: Part 3 (full corpus) v 2.0 (mpg+ syntactic analysis). Linguistic Data Consortium, Philadelphia. find
  19. Maamouri, M., Graff, D., Bouziri, B., Krouna, S., Bies, A., and Kulick, S. (2010). Standard arabic morphological analyzer (sama) version 3.1. Linguistic Data Consortium, Catalog No.: LDC2010L01. find
  20. Monroe, W., Green, S., and Manning, C. D. (2014). Word segmentation of informal arabic with domain adaptation. ACL, Short Papers.
  21. Pasha, A., Al-Badrashiny, M., Diab, M., El Kholy, A., Eskander, R., Habash, N., Pooleery, M., Rambow, O., and Roth, R. M. (2014). Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In LREC-2014, Reykjavik, Iceland.
  22. Sajjad, H., Guzmn, F., Nakov, P., Abdelali, A., Murray, K., Obaidli, F. A., and Vogel, S. (2013). QCRI at IWSLT 2013: Experiments in Arabic-English and English-Arabic spoken language translation. In Proceedings of the 10th International Workshop on Spoken Language Technology (IWSLT-13), December. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/darwish-2016-farasa-new-fast-accurate-arabic/