Elamur Research⌕
☰
home / papers / 2022 / Towards Universal Segmentations: UniSegments 1.0

Towards Universal Segmentations: UniSegments 1.0

Summary

Plenty of annotated data already exists for morphological segmentation, that is, dividing a word form into morphs. However, each resource has its own format, understands the morph in its own way, and segments words with different degrees of completeness. There is no common standard like Universal Dependencies for this task. The authors from Charles University examined 17 existing resources for 32 languages. They compared how these resources represent allomorphy, homonymy, interfixes, and the difference between partial and complete segmentation. On this basis, a unified scheme is proposed. In it, each word is described by a list of segments, and each segment has character positions (span), a type (root, prefix, suffix, interfix, ending, etc.) and, where possible, the corresponding morpheme. All resources were converted into this scheme by automatic converters; no new manual annotation was added. Missing data, such as morph boundaries given a known sequence of morphemes, were reconstructed using heuristics. Harmonized versions of the freely licensed resources have been published as the UniSegments 1.0 collection. This is a step toward giving segmentation the same kind of common standard that UD has become for syntax.

Key points

  • 17 resources and 32 languages are covered. Among them are DeriNet (Czech), DErivBase (German), DerivBase.Ru, Démonette, Échantinom, DerIvaTario, MorphoLex, Word Formation Latin, MorphyNet (15 languages), Uniparser (7 languages, including Erzya, Udmurt and Tajik) and KCIS (Hindi, Marathi, Malayalam, Kannada, Bengali).
  • The format is a JSON record for each word with a segmentation field. Each segment has a list of character indices (span) and a type: root, prefix, suffix, interfix, ending or unsegmented. Attributes specific to the source resource are stored alongside.
  • Some resources were converted but not published because of non-free licenses. Among them are CELEX (Dutch, English, German) and the Russian morphemic dictionaries by Kuznetsova–Efremova and Tikhonov.
  • Arabic data from MorphoChallenge were not included because morphs could not be aligned with morphemes.
  • Ukrainian is not in the collection. The closest languages with data are Russian, Czech, Polish and Croatian.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zabokrtsku2022universal,
  title = {{Towards Universal Segmentations: UniSegments 1.0}},
  author = {Žabokrtský, Zdeněk and Bafna, Niyati and Bodnár, Jan and Kyjánek, Lukáš and Svoboda, Emil and Ševčíková, Magda and Vidra, Jonáš},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/3ukt8xv5nadt},
  url = {https://doi.org/10.63317/3ukt8xv5nadt}
}

References

55 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Angle, S., Rao, B. A., and Muralikrishna, S. (2018). Kannada morpheme segmentation using machine learning. International Journal of Engineering and Technology (UAE), 7(2):45–49. find
  2. Arkhangelskiy, T. and Medvedeva, M. (2016). Developing morphologically annotated corpora for minority languages of Russia. In Sandra Kübler et al., editors, Proceedings of Corpus Linguistics Fest 2016 (CLiF 2016), volume 1607 of CEUR Workshop Proceedings, pages 1–6, Bloomington, Indiana, USA, June. find
  3. CEUR-WS.org. Arkhangelskiy, T., Belyaev, O., and Vydrin, A. (2012). The creation of large-scale annotated corpora of minority languages using UniParser and the EANC platform. In Proceedings of COLING 2012: Posters, pages 83–92, Mumbai, India, December. find
  4. The COL-ING 2012 Organizing Committee. find
  5. Arkhangelskiy, T. (2019). Corpora of social media in minority Uralic languages. In Proceedings of the Fifth International Workshop on Computational Linguistics for Uralic Languages, pages 125–140, Tartu, Estonia, January. Association for Computational Linguistics. find
  6. Aronoff, M. and Fudeman, K. (2011). What is Morphology? Wiley-Blackwell, Malden, MA, 2nd edition. find
  7. Aronoff, M. (1976). Word Formation in Generative Grammar. find
  8. The MIT Press, Cambridge. Atmakuri, S., Shahi, B., Rao, B. A., and Muralikrishna, S. (2018). A comparison of features for pos tagging in kannada. International Journal of Engineering and Technology (UAE), 7(4):2418–2421. find
  9. Balota, D. A., Yap, M. J., Hutchison, K. A., Cortese, M. J., Kessler, B., Loftis, B., Neely, J. H., Nelson, D. L., Simpson, G. B., and Treiman, R. (2007). The English lexicon project. Behavior Research Methods, 39(3):445–459, Aug. find
  10. Batsuren, K., Bella, G., and Giunchiglia, F. (2021). MorphyNet: a large multilingual database of derivational and inflectional morphology. In Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 39–48, Online, August. Association for Co… find our card
  11. Bertinetto, P. M., Burani, C., Laudanna, A., Mar-coni, L., Ratti, D., Rolando, C., and Thornton, A. M. (2005). Colfis (corpus e lessico di frequenza dell’italiano scritto). Available on http://www.istc. cnr. it/material/database, pages 67–73. ↗
  12. Bharati, A., Sangal, R., Sharma, D. M., and Bai, L. (2006). Anncorra: Annotating corpora guidelines for POS and chunk annotation for Indian languages. LTRC-TR31, pages 1–38. find
  13. Bodnár, J., Žabokrtský, Z., and Ševčíková, M. (2020). Semi-supervised induction of morpheme boundaries in czech using a word-formation network. In 23rd International Conference on Text, Speech and Dialogue, pages 189–196, Cham, Switzerland. find
  14. Springer. Bonami, O., Caron, G., and Plancq, C. (2014). Construction d’un lexique flexionnel phonétisé libre du français. In SHS Web of Conferences, volume 8, pages 2583–2596. find
  15. EDP Sciences. Cotterell, R., Vieira, T., and Schütze, H. (2016). A joint model of orthography and morphological segmentation. In Proceedings of NAACL. Association for Computational Linguistics. de Marneffe, M.-C., Manning, C., Nivre, J., and Zeman, D. (2021). Universal Dependencies. Computational Li… find
  16. Ferrand, L., New, B., Brysbaert, M., Keuleers, E., Bonin, P., Méot, A., Augustinova, M., and Pallier, C. (2010). The French lexicon project: Lexical decision data for 38,840 french words and 38,840 pseu-dowords. Behavior Research Methods, 42(2):488– 496, May. find
  17. Gala, N. and Rey, V. (2008). POLYMOTS : une base de données de constructions dérivationnelles en français à partir de radicaux phonologiques. In Actes de la 15ème conférence sur le Traitement Automatique des Langues Naturelles. Articles courts, pages 91–100, Avignon, France, June. find
  18. ATALA. Haspelmath, M. and Sims, A. D. (2010). find
  19. Understanding Morphology. Hodder Education, London. Hathout, N. and Namer, F. (2014). Démonette, a French Derivational Morpho-Semantic Network. Linguistic Issues in Language Technology, 11:125– 162. find
  20. Hathout, N., Namer, F., and Dal, G. (2002). An Experimental Constructional Database: The MorTAL Project. In Paul Boucher, editor, Many Morphologies, pages 178–209. find
  21. Cascadilla, Somerville, Mass. Hathout, N. (2011). Morphonette: a paradigm-based morphological network. Lingue e linguaggio, 10(2):245–264. find
  22. Iskandarova, D. M. (2021). Национальный корпус таджикского языка как инструмент лингвистических исследований (National corpus of the Tajik language as a tool for linguistic research). Kazan Science, 1:94–97. find
  23. Khurshudian, V. and Daniel, M. (2009). Eastern Armenian national corpus. Dialog’2009, pages 509– 518. find
  24. Kurimo, M., Virpioja, S., Turunen, V., and Lagus, K. (2010). Morpho challenge competition 2005–2010: Evaluations and results. In Proceedings of the 11th Meeting of the ACL Special Interest Group on Computational Morphology and Phonology, SIGMOR-PHON ’10, page 87–95, USA. Association for Computationa… find
  25. Kuznetsova, A. I. and Efremova, T. F. (1986). Slovar’ morfem russkogo jazyka [Dictionary of morphemes of the Russian language]. Russkij jazyk, Moscow. find
  26. Litta, E., Passarotti, M., and Culy, C. (2016). Formatio formosa est. Building a word formation lexicon for Latin. In Proceedings of Third Italian Conference on Computational Linguistics (CLiC-it). find
  27. Mailhot, H., Wilson, M. A., Macoir, J., Deacon, S. H., and Sánchez-Gutiérrez, C. H. (2020). MorphoLex-FR: A derivational morphological database for 38,840 French words. Behavior Research Methods, 52(3):1008–1025, June. find
  28. McCarthy, A. D., Kirov, C., Grella, M., Nidhi, A., Xia, P., Gorman, K., Vylomova, E., Mielke, S. J., Nicolai, G., Silfverberg, M., et al. (2020). Unimorph 3.0: Universal morphology. In Proceedings of the 12th Language Resources and Evaluation Conference. European Language Resources Association (ELRA… find our card
  29. Namer, F. (2009). Morphologie, lexique et traitement automatique des langues. find
  30. Hermès-Lavoisier. New, B., Brysbaert, M., Veronis, J., and Pallier, C. (2007). The use of film subtitles to estimate word frequencies. Applied psycholinguistics, 28(4):661– 677. find
  31. Petrov, S., Das, D., and McDonald, R. (2012). A universal part-of-speech tagset. In Nicoletta Calzolari, et al., editors, Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey, may. European Language Resources Association (ELRA). find
  32. Porter, M. F. (1980). An algorithm for suffix stripping. find
  33. Program. Rajman, M., Lecomte, J., and Paroubek, P. (1997). Format de description lexicale pour le français, partie 2: Description morpho-syntaxique. Technical Report GRACE GTR-3-2.1, EPFL & INaLF. find
  34. Redkar, H., Singh, S., Ghag, N., Paranjape, J., Joshi, N., Kulkarni, M., and Bhattacharyya, P. (2016). Verbframator: Semi-automatic verb frame annotator tool with special reference to marathi. In Proceedings of the 13th International Conference on Natural Language Processing, pages 299–304. find
  35. Santorini, B. (1990). Part-of-speech tagging guidelines for the Penn Treebank Project. Technical Report MS-CIS-90-47, Department of Computer and Information Science, University of Pennsylvania, jan. find
  36. Sennrich, R., Haddow, B., and Birch, A. (2016). Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725. find
  37. Smit, P., Virpioja, S., Grönroos, S.-A., Kurimo, M., et al. (2014). Morfessor 2.0: Toolkit for statistical morphological segmentation. In The 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL), Gothenburg, Sweden, April 26-30, 2014. find
  38. Aalto University. Šojat, K., Srebačić, M., Pavelić, T., and Tadić, M. (2014). CroDeriV: A New Resource for Processing Croatian Morphology. In Proceedings of the Language Resources and Evaluation (LREC-2014), volume 14, pages 3366–3370, Reykjavik. Citeseer. find
  39. Sokolová, M., Ološtiak, M., Ivanová, M., Šimon, F., Czéreová, B., Vužňáková, K., Benko, V., and Moško, G. (2005). Slovník koreňovỳch morfém slovenčiny. Prešovská univerzita. find
  40. Sánchez-Gutiérrez, C. H., Mailhot, H., Deacon, S. H., and Wilson, M. A. (2018). MorphoLex: A derivational morphological database for 70,000 English words. Behavior Research Methods, 50(4):1568– 1580, August. find our card
  41. Talamo, L., Celata, C., and Bertinetto, P. M. (2016). DerIvaTario: An Annotated Lexicon of Italian Derivatives. Word Structure, 9(1):72–102. find
  42. Tandon, J. and Sharma, D. M. (2017). Unity in diversity: A unified parsing strategy for major indian languages. In Proceedings of the Fourth International Conference on Dependency Linguistics (Depling 2017), pages 255–265. find
  43. Tanguy, L. and Hathout, N. (2002). Webaffix : un outil d’acquisition morphologique dérivationnelle à partir du Web. In Jean-Marie Pierrel, editor, Actes de la 9ème Conférence Annuelle sur le Traitement Automatique des Langues Naturelles (TALN-2002), pages 245–254, Nancy. find
  44. ATALA. Tikhonov, A. N. (1996). Морфемно-орфографический словарь русского языка. Русская морфемика (Morphemic-spelling dictionary of the Russian language. Russian morphemics). find
  45. Shkola-Press, Moscow, Russia. Todi, K. K., Mishra, P., and Misra Sharma, D. (2018). Building a kannada pos tagger using machine learning and neural network models. arXiv e-prints, pages arXiv–1808. find
  46. Vodolazsky, D. (2020). DerivBase.Ru: A derivational morphology resource for Russian. In Proceedings of the Language Resources and Evaluation (LREC-2020), volume 20, pages 3930–3936, Marseille, France. find
  47. Zeller, B., Šnajder, J., and Padó, S. (2013). DErivBase: Inducing and evaluating a derivational morphology resource for German. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, volume 1, pages 1201–1211. ACL. \#\# 9. find
  48. Language Resource References Ansari, E., Žabokrtský, Z., Haghdoost, H., and Nikravesh, M. (2019). Persian Morphologically Segmented Lexicon 0.5. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find
  49. Baayen, R. Harald and Piepenbrock, Richard and Gulikers, Leon. (1995). The CELEX Lexical Database (CD-ROM). Linguistic Data Consortium, ISLRN 204-698-863-053-1. find
  50. Bhat, R. A., Bhatt, R., Farudi, A., Klassen, P., Narasimhan, B., Palmer, M., Rambow, O., Sharma, D. M., Vaidya, A., Vishnu, S. R., et al. (2017). The hindi/urdu treebank project. In Handbook of Linguistic Annotation, pages 659–697. find
  51. Springer. Bonami, O. and Tribout, D. (2021). find
  52. Echantinom. Hajič, Jan and Hlaváčová, Jaroslava and Mikulová, Marie and Straka, Milan and Štěpánková, Barbora. (2020). MorfFlex CZ 2.0. Institute of Formal and Applied Linguistics, LINDAT/CLARIN, Charles University. find
  53. Kyjánek, L., Žabokrtský, Z., Vidra, J., and Ševčíková, M. (2021). Universal derivations v1.1. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find
  54. Rao, A. B., Murali Krishna, S., and Nayak, A. (2014). Developing a dependency treebank for kannada. International Journal of Engineering Sciences and Research, 15:5–7. find
  55. Vidra, J., Žabokrtský, Z., Kyjánek, L., Ševčíková, M., Dohnalová, Š., Svoboda, E., and Bodnár, J. (2021). DeriNet 2.1. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/abokrtsky-2022-universal-segmentations-unisegments-1-0/