Elamur Research⌕
☰
home / papers / 2014 / Morfessor 2.0: Toolkit for statistical morphological segmentation

Morfessor 2.0: Toolkit for statistical morphological segmentation

Summary

Morfessor 2.0 is a re-implementation of the Morfessor Baseline algorithm, a probabilistic method for unsupervised morphological segmentation. The software splits words into morph-like units using raw text alone. The original Morfessor 1.0 had stayed in wide use for speech recognition, machine translation and speech retrieval, so the authors rebuilt it as a modular, openly licensed Python toolkit with command-line tools and a library API. The model is still trained by MAP estimation: a lexicon prior that prefers fewer and shorter units is balanced against the data likelihood. Training uses a recursive binary-splitting local search, and Viterbi decoding segments new words. The new release adds semi-supervised learning with weighted annotated data, on-line training, n-best Viterbi segmentation, and built-in evaluation with boundary precision/recall (BPR) and significance testing. The code is also generic: it treats any sequence of atoms as segmentable data, not just words. This is a demonstration paper and reports no benchmark numbers. Its contribution is a maintained, extensible tool for morphologically rich, concatenative languages.

Key points

  • Morfessor 2.0 rewrites Morfessor Baseline (Creutz and Lagus, 2002), whose first implementation was Morfessor 1.0 (2005). It is released open source under a permissive license on GitHub (aalto-speech/morfessor) and the Morpho project site.

  • Semi-supervised training follows Kohonen et al. (2010):

    • A weight α on the corpus likelihood controls over- versus under-segmentation.
    • A separate weight β scales the likelihood of the annotated data.
    • Both weights can be grid-searched on development data, or set with built-in heuristics.

  • Morfessor 2.0 changes the model in two ways compared with 1.0. It adds compound-boundary probabilities to the likelihood, and it supports only an implicit exponential length prior.

  • The workflow uses three command-line tools: morfessor-train, morfessor-segment and morfessor-evaluate. Evaluation reports BPR (Virpioja et al., 2011) and includes a Wilcoxon signed-rank test for comparing models.

  • The web demo covers six languages:

    LanguageRunning wordsWord forms
    English62M~385K
    Estonian212M~3.9M
    Finnish36M~2.2M
    German46M~1.27M
    Swedish1M~92K
    Turkish12M~617K

    For each word, the demo shows the 5 best segmentations.

Interesting ideas

  • I1. Domain-neutral vocabulary. The code talks about "atoms", "compounds" and "constructions" instead of letters, words and morphs. The same segmenter can therefore be reused for other tasks, such as chunking sentences into phrases or splitting other symbol sequences.
  • I2. Separate weight for scarce labeled data. A small annotated set can be swamped by the likelihood of a huge raw corpus. Giving it its own scaling factor is a simple, general fix for any semi-supervised objective that mixes small gold data with large unlabeled data.
  • I3. Tuning a hyperparameter during training. α is adjusted on the fly to balance boundary precision against recall, which avoids a full grid search. Similar heuristics could tune other segmenters, such as subword tokenizers for Ukrainian NLP.
  • I4. Random skipping of recently seen items. In on-line or batch training, frequent words that were just analyzed are probabilistically skipped. This saves compute on Zipfian text data with little cost to the model.
  • I5. Evaluation and significance testing built into the tool. Shipping BPR scoring and a Wilcoxon test with the toolkit makes model comparisons reproducible. Returning n-best segmentations with their probabilities also exposes the model's uncertainty for downstream use.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics Morphology

 Institutions

🎓Aalto UniversityEspoo · Finland4 works · 559 citations

Cite

Cite as Peter Smit, Sami Virpioja, Stig-Arne Grönroos and Mikko Kurimo. 2014. Morfessor 2.0: Toolkit for statistical morphological segmentation. Proceedings of the Demonstrations at the 14th Conference of the European Chapter of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/e14-2006

@inproceedings{smit2014morfessor,
  title = {{Morfessor 2.0: Toolkit for statistical morphological segmentation}},
  author = {Peter Smit, Sami Virpioja, Stig-Arne Grönroos and Mikko Kurimo},
  booktitle = {Proceedings of the Demonstrations at the 14th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2014},
  doi = {10.3115/v1/e14-2006},
  url = {https://doi.org/10.3115/v1/e14-2006}
}

References

15 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. E. Arisoy, D. Can, S. Parlak, H. Sak, and M. Saraclar. 2009. Turkish broadcast news transcription and retrieval. Audio, Speech, and Language Processing, IEEE Transactions on, 17(5):874–883. find
  2. M. Creutz and K. Lagus. 2002. Unsupervised discovery of morphemes. In Mike Maxwell, editor, Proceedings of the ACL-02 Workshop on Morphological and Phonological Learning, pages 21–30. Association for Computational Linguistics, July. find
  3. M. Creutz and K. Lagus. 2005a. Inducing the morphological lexicon of a natural language from unannotated text. In Proceedings of AKRR’05, International and Interdisciplinary Conference on Adaptive Knowledge Representation and Reasoning, pages 106–113, Espoo, Finland, June. Helsinki University of Tec… find
  4. M. Creutz and K. Lagus. 2005b. Unsupervised morpheme segmentation and morphology induction from text corpora using Morfessor 1.0. Technical Report A81, Publications in Computer and Information Science, Helsinki University of Technology. find
  5. A. El-Desoky Mousa, M. Ali Basha Shaik, R. Schluter, and H. Ney. 2010. Sub-lexical language models for German LVCSR. In Spoken Language Technology Workshop (SLT), 2010 IEEE, pages 171–176. IEEE. find
  6. T. Hirsimäki, M. Creutz, V. Siivola, M. Kurimo, S. Virpioja, and J. Pylkkönen. 2006. Unlimited vocabulary speech recognition with morph language models applied to Finnish. Computer Speech & Language, 20(4):515–541. find
  7. T. Hirsimäki, J. Pylkkönen, and M. Kurimo. 2009. Importance of high-order n-gram models in morph-based speech recognition. Audio, Speech, and Language Processing, IEEE Transactions on, 17(4):724–732. find
  8. O. Kohonen, S. Virpioja, and K. Lagus. 2010. Semi-supervised learning of concatenative morphology. In Proceedings of the 11th Meeting of the ACL Special Interest Group on Computational Morphology and Phonology, pages 78–86, Uppsala, Sweden, July. Association for Computational Linguistics. find
  9. C. Monson, K. Hollingshead, and B. Roark. 2010. Simulating morphological analyzers with stochastic taggers for confidence estimation. In Multilingual Information Access Evaluation I. Text Retrieval Experiments, pages 649–657. Springer. find
  10. S. Spiegler, B. Golénia, K. Shalonova, P. Flach, and R. Tucker. 2008. Learning the morphology of zulu with different degrees of supervision. In Spoken Language Technology Workshop, 2008. find
  11. SLT 2008. IEEE, pages 9–12. IEEE. find
  12. S. Virpioja, J. Väyrynen, M. Creutz, and M. Sadeniemi. 2007. Morphology-aware statistical machine translation based on morphs induced in an unsupervised manner. In Proceedings of the Machine Translation Summit XI, pages 491–498, Copenhagen, Denmark, September. find
  13. S. Virpioja, O. Kohonen, and K. Lagus. 2010. Unsupervised morpheme analysis with Allomorfessor. In Multilingual Information Access Evaluation I. Text Retrieval Experiments, volume 6241 of LNCS, pages 609–616. Springer Berlin / Heidelberg. find
  14. S. Virpioja, V. Turunen, S. Spiegler, O. Kohonen, and M. Kurimo. 2011. Empirical comparison of evaluation methods for unsupervised learning of morphology. TAL, 52(2):45–90. find
  15. S. Virpioja, P. Smit, S. Grönroos, and M. Kurimo. 2013. Morfessor 2.0: Python implementation and extensions for Morfessor Baseline. Report 25/2013 in Aalto University publication series SCIENCE + TECHNOLOGY, Aalto University, Finland. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/smit-2014-morfessor-2-0-toolkit-statistical/