Elamur Research⌕
☰
home / papers / 2019 / Investigating Machine Learning Methods for Language and Dialect Identification of Cuneiform Texts

Investigating Machine Learning Methods for Language and Dialect Identification of Cuneiform Texts

Summary

Telling apart languages and dialects written in cuneiform is hard. Digital resources are scarce, and no general tokenizer exists, because the script mixes syllabic and logographic signs. For the VarDial 2019 Cuneiform Language Identification (CLI) shared task, the SharifCL team classified text samples into seven classes: Sumerian and six Akkadian dialects. To avoid segmentation entirely, they used only character-level input. They compared several classifiers built in Scikit-learn on character n-gram TF-IDF and count features:

  • random forest
  • logistic regression
  • multinomial naive Bayes
  • SVM
  • a soft-voting ensemble of the SVM and naive Bayes

They also tested CNN and LSTM models over character embeddings. Each method was judged on macro F1, accuracy and training time. The SVM + naive Bayes ensemble, retrained on train plus development data, reached a macro F1 of 72.10% on the test set. It trained in seconds, while the neural models needed over an hour. The work shows that cheap linear models on character features give a solid baseline for low-resource scripts. Some closely related dialects still remain poorly separated.

Key points

  • The CLI data has 7 classes: SUX, OLB, MPB, STB, NEB, LTB and NEA.

    • There are 139,421 training samples, and the classes are heavily skewed. SUX holds 38.49%, while OLB has only 3,803 samples (2.72%).
    • The development set has 668 samples per label and the test set has 985 per label.

  • Features were 1–3-gram character TF-IDF plus 1–4-gram character counts.

    • The neural models instead used 32-dimensional embeddings of the 256 most frequent characters.

  • Development macro F1 by model, with training time:

    ModelDev macro F1Training time
    Ensemble0.72683.34 s
    SVM0.72221.67 s
    NB0.71940.15 s
    LR0.6861not given here
    RNN0.6259over 4,000 s
    CNN0.6192over 4,000 s
    RF0.5201264 s

  • Test macro F1 by model and training data:

    • SVM trained on train only: 0.6660
    • SVM trained on train plus dev: 0.7171
    • Ensemble trained on train plus dev: 0.7210 (accuracy 0.7239)

  • Per-class test results show uneven difficulty.

    • LTB was easiest, with F1 0.9269 and recall 96.55%.
    • STB was hardest, with F1 0.5214; it was often confused with SUX.
    • NEB was frequently labelled STB.
    • NEA had high recall (87.72%) but low precision (0.5621).

Interesting ideas

  • I1. When reliable word boundaries cannot be produced, classifying on overlapping character n-grams sidesteps segmentation and still captures orthographic and morphological cues.
  • I2. Soft voting sums the class probabilities of two different cheap models (a margin-based SVM and a generative naive Bayes) and gives a small but consistent gain over either alone.
  • I3. Reporting training time next to F1 shows that a model several thousand times cheaper can match or beat neural ones on small data.
  • I4. Per-class precision/recall and a confusion matrix expose failure modes that macro F1 hides, such as a class that is over-predicted (high recall, low precision) and systematic confusions between neighbouring dialects.
  • I5. For the final submission, the authors folded the development set into training once hyperparameters were fixed, which noticeably raised test scores.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): a character n-gram side-model could flag the variety or register of input text before the tokenizer, and could guess tags for out-of-vocabulary strings where word rules fail.
  • I2 (learn): when several witness parsers or taggers agree on silver data, combining their confidence scores rather than hard votes could give a more stable consensus label.
  • I3 (learn): evaluation reports for parser and tagger variants could list training and inference cost alongside accuracy, to justify keeping the explainable perceptron path where it is competitive.
  • I4 (learn): per-label confusion tables for UD relations or frame roles would show which neighbouring categories get swapped and where repair rules are most needed.
  • I5 (tree): after tuning on a held-out split, retraining the parser on all treebank data, including dev, before release could add accuracy cheaply.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Akkadian and cuneiform NLP

 Institutions

🎓Sharif University of TechnologyTehran · Iran2 works · 524 citations

Cite

Cite as Ehsan Doostmohammadi and Minoo Nassajian. 2019. Investigating Machine Learning Methods for Language and Dialect Identification of Cuneiform Texts. Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-1420

@inproceedings{doostmohammadi2019investigating,
  title = {{Investigating Machine Learning Methods for Language and Dialect Identification of Cuneiform Texts}},
  author = {Ehsan Doostmohammadi and Minoo Nassajian},
  booktitle = {Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects},
  year = {2019},
  doi = {10.18653/v1/w19-1420},
  url = {https://doi.org/10.18653/v1/w19-1420}
}

References

43 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Ali, Najim Dehak, Patrick Cardinal, Sameer Khurana, Sree Harsha Yella, James Glass, Peter Bell, and Steve Renals. 2015. Automatic dialect detection in arabic broadcast speech. arXiv preprint arXiv:1509.06928. ↗
  2. Khaled Alrifai, Ghaida Rebdawi, and Nada Ghneim. 2017. Arabic tweeps gender and dialect prediction. In CLEF (Working Notes). NLP for Similar Languages, Varieties and Dialects (VarDial), pages 170–177. find
  3. Adrien Barbaresi. 2016. An unsupervised morphological criterion for discriminating similar languages. In 3rd Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2016), pages 212–220. Association for Computational Linguistics. find
  4. Bernardt Duvenhage, Mfundo Ntini, and Phala Ra-monyai. 2017. Improved text language identification for the south african languages. In 2017 Pattern Recognition Association of South Africa and Robotics and Mechatronics (PRASA-RobMech), pages 214–218. IEEE. find
  5. François Barthélemy. 1998. A morphological analyzer for akkadian verbal forms with a model of phonetic transformations. In Proceedings of the Workshop on Computational Approaches to Semitic Languages, pages 73–81. Association for Computational Linguistics. our card
  6. Mahmoud El Haj, Paul Edward Rayson, and Mariam Aboelezz. 2018. Arabic dialect identification in the context of bivalency and code-switching. find
  7. Marc Franco-Salvador, Greg Kondrak, and Paolo Rosso. 2017. Bridging the native language and language variety identification tasks. Procedia computer science, 112:1554–1561. find
  8. François Barthélemy. 2009. The karamel system and semitic languages: structured multi-tiered morphology. In Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages, pages 10–18. Association for Computational Linguistics. our card
  9. Helena Gomez, Ilia Markov, Jorge Baptista, Grigori Sidorov, and David Pinto. 2017. Discriminating between similar languages using a combination of typed and untyped character n-grams and words. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (Var-Dial), pages 137–145. find
  10. Manas Jyoti Bora and Ritesh Kumar. 2018. Automatic word-level identification of language in assamese english hindi code-mixed data. In 4th Workshop on Indian Language Data and Resources, Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 7–12. find
  11. Cyril Goutte, Serge Léger, Shervin Malmasi, and Marcos Zampieri. 2016. Discriminating similar languages: Evaluations and explorations. arXiv preprint arXiv:1610.00031. ↗
  12. Ralf D Brown. 2012. Finding and identifying text in 900+ languages. Digital Investigation, 9:S34–S43. find
  13. Abualsoud Hanani, Aziz Qaroush, and Stephen Taylor. 2017. Identifying dialects with textual and acoustic cues. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), pages 93–101. find
  14. Dayvid W Castro, Ellen Souza, Douglas Vitório, Diego Santos, and Adriano LI Oliveira. 2017. Smoothed n-gram based models for tweet language identification: A case study of the brazilian and european portuguese national varieties. Applied Soft Computing, 61:1160–1172. find
  15. Andrea Hategan, Bogdan Barliga, and Ioan Tabus. 2009. Language identification of individualwords in a multilingual automatic speech recognition system. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 4357– 4360. IEEE. find
  16. Jordan Cazamias, Chinmayi Dixit, and Martina Marek. 2015. Large-scale language classification. find
  17. Joseph Chee Chang and Chu-Cheng Lin. 2014. Recurrent-neural-network for language detection on twitter code-switching corpus. arXiv preprint arXiv:1412.4314. ↗
  18. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
  19. Christian Chiarcos, Ilya Khait, Émilie Pagé-Perron, and Maria Sukhareva. 2017. Machine translation and automated analysis of the sumerian language. In Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, ACL, pages 10–16. Association for Computational Linguistics. find
  20. Timo Homburg and Christian Chiarcos. 2016. Akkadian word segmentation. In Proceedings Tenth International Conference on Language Resource Evaluation.(LREC 2016), pages 4067–4074. find
  21. Tommi Jauhiainen, Heidi Jauhiainen, Tero Alstola, and Krister Lindén. 2019. Language and Dialect Identification of Cuneiform Texts. In Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  22. Christian Chiarcos, Emilie Page-Perron, Niko Schenk, Jayanth, and Lucas Reckling. 2018. Annotating sumerian: A llod-enhanced workflow for cuneiform corpora. In Proceedings of the Language Resources and Evaluation Conference. find
  23. Tommi Jauhiainen, Krister Lindén, and Heidi Jauhiainen. 2017. Evaluation of language identification methods using 285 languages. In Proceedings of the 21st Nordic Conference on Computational Linguistics, pages 183–191. find
  24. Simon Clematide and Peter Makarov. 2017. Cluzh at vardial gdi 2017: Testing a variety of machine learning tools for the classification of swiss german dialects. In Proceedings of the Fourth Workshop on Francisco Rangel, Marc Franco-Salvador, and Paolo Rosso. 2016. A low dimensionality representation for language variety identification. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 156–169. Springer. find
  25. Tommi Jauhiainen, Marco Lui, Marcos Zampieri, Timothy Baldwin, and Krister Lindén. 2018. Automatic language identification in texts: A survey. arXiv preprint arXiv:1804.08186. ↗ our card
  26. David Jurgens, Yulia Tsvetkov, and Dan Jurafsky. 2017. Incorporating dialectal variability for socially equitable language identification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 51–57. find
  27. Younes Samih and Laura Kallmeyer. 2017. Dialectal Arabic processing Using Deep Learning. Ph.D. thesis, Ph. D. thesis, Düsseldorf, Germany. find
  28. Anil Kumar Singh and Jagadeesh Gorla. 2007. Identification of languages and encodings in a multilingual document. In Building and Exploring Web Corpora (WAC3-2007): Proceedings of the 3rd Web as Corpus Workshop, Incorporating Cleaneval, volume 4, page 95. Presses univ. de Louvain. find
  29. Laura Kataja and Kimmo Koskenniemi. 1988. Finite-state description of semitic morphology: A case study of ancient accadian. In Coling Budapest 1988 Volume 1: International Conference on Computational Linguistics, volume 1. our card
  30. Valentin Tablan, Wim Peters, Diana Maynard, Hamish Cunningham, and K Bontcheva. 2006. Creating tools for morphological analysis of sumerian. In LREC, pages 1762–1765. find
  31. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  32. Jennifer Williams and Charlie Dagli. 2017. Twitter language identification of similar languages and dialects without ground truth. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), pages 73–83. find
  33. Tom Kocmi and Ondřej Bojar. 2017. Lanidenn: Multilingual language identification on character window. arXiv preprint arXiv:1701.03338. ↗
  34. Chris van der Lee and Antal van den Bosch. 2017. Exploring lexical and syntactic features for language variety identification. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), pages 190–199. find
  35. Kenji Yamauchi, Hajime Yamamoto, and Wakaha Mori. 2018. Building a handwritten cuneiform character imageset. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). find
  36. Liang Luo, Yudong Liu, James Hearne, and Clinton Burkhart. 2015. Unsupervised sumerian personal name recognition. In The Twenty-Eighth International Flairs Conference. find
  37. Marcos Zampieri, Shervin Malmasi, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, Jörg Tiedemann, Yves Scherrer, and Noëmi Aepli. 2017. Findings of the VarDial Evaluation Campaign 2017. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), Valencia, Spain. our card
  38. Aaron Macks. 2002. Parsing akkadian verbs with prolog. In Proceedings of the ACL-02 workshop on Computational approaches to semitic languages. our card
  39. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Ahmed Ali, Suwon Shuon, James Glass, Yves Scherrer, Tanja Samardžić, Nikola Ljubešić, Jörg Tiedemann, Chris van der Lee, Stefan Grondelaers, Nelleke Oostdijk, Antal van den Bosch, Ritesh Kumar, Bornini Lahiri, and Mayank Jain. 2018. Language Identification and Morphosyntactic Tagging: The Second VarDial Evaluation Campaign. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), Santa Fe, USA. our card
  40. Shervin Malmasi and Marcos Zampieri. 2016. Arabic dialect identification in speech transcripts. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 106–113. find
  41. Shervin Malmasi, Marcos Zampieri, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, and Jörg Tiedemann. 2016. Discriminating between similar languages and arabic dialect identification: A report on the third dsl shared task. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 1–14. our card
  42. Marcos Zampieri, Shervin Malmasi, Yves Scherrer, Tanja Samardžić, Francis Tyers, Miikka Silfverberg, Natalia Klyueva, Tung-Le Pan, Chu-Ren Huang, Radu Tudor Ionescu, Andrei Butnaru, and Tommi Jauhiainen. 2019. A Report on the Third VarDial Evaluation Campaign. In Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). Association for Computational Linguistics. find
  43. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine learning in python. J. Mach. Learn. Res., 12:2825–2830. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/doostmohammadi-2019-investigating-machine-learning-methods-language/