Elamur Research⌕
☰
home / papers / 2015 / Modeling Word Forms Using Latent Underlying Morphs and Phonology

Modeling Word Forms Using Latent Underlying Morphs and Phonology

Paper · 2015 · Ryan Cotterell, Nanyun Peng and Jason Eisner

Summary

The paper addresses a classic phonology puzzle: how to automatically infer the single abstract "underlying" form of each morpheme, and the sound rules that turn concatenated morphemes into the pronunciations actually observed. Linguists usually solve this by hand. The authors treat it as joint inference in a directed graphical model (a Bayesian network) whose variables are unknown strings. A lexicon model, a probabilistic finite-state automaton, generates morph URs. These are joined with boundary symbols into word URs. A stochastic phonology, encoded as a weighted finite-state transducer with trainable weights, then maps each word UR to its surface form. Inference uses loopy belief propagation over weighted finite-state machines, so candidate URs are not restricted to a small preselected set. Parameters are tuned with MAP-EM. The input is surface word types paired with their abstract morpheme sequences. The authors also propose an evaluation setup based on predicting held-out surface forms. Across subsets of 7 languages, the learner predicted unseen forms with low error. This matters for generating and analyzing novel words in morphologically rich languages and for comparing phonological theories.

Key points

  • The model has four layers: morpheme URs, concatenated word URs (joined by #), word SRs produced by the phonology, and observations. In the paper's experiments, the SR and observation layers are merged.
  • The phonology S_θ is a contextual edit process implemented as a probabilistic finite-state transducer. It uses roughly 50,000 features, and exceptions are handled as low-probability edits.
  • Inference relies on loopy BP: finite-state "messages" score every possible string. Learning uses EM, which may get stuck in local optima.
  • The task requires observed morpheme annotation. Unlike Dreyer and Eisner's paradigm models, this directed model is productive and can generate unboundedly many new words, including derivational ones.
  • Evaluation covers 7 languages on both small and larger datasets, using a newly defined held-out SR prediction paradigm.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States240 works · 96,487 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Morphology · 405 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Word form · 16 works

Cite

Cite as Ryan Cotterell, Nanyun Peng and Jason Eisner. 2015. Modeling Word Forms Using Latent Underlying Morphs and Phonology. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00149

@article{cotterell2015modeling,
  title = {{Modeling Word Forms Using Latent Underlying Morphs and Phonology}},
  author = {Ryan Cotterell, Nanyun Peng and Jason Eisner},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.1162/tacl\_a\_00149},
  url = {https://doi.org/10.1162/tacl_a_00149}
}

References

62 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Diana Apoussidou. 2006. On-line learning of underlying forms. Technical Report ROA-835, Rutgers Optimality Archive. find
  2. R. Harald Baayen, Richard Piepenbrock, and Leon Gulikers. 1995. The CELEX lexical database on CD-ROM. find
  3. Juliette Blevins. 1994. A phonological and morphological reanalysis of the Maori passive. Te Reo, 37:29–53. find
  4. George N. Clements and Elizabeth V. Hume. 1995. The internal organization of speech sounds. In John Goldsmith, editor, Handbook of Phonological Theory. Oxford University Press, Oxford. find
  5. Ryan Cotterell and Jason Eisner. 2015. Penalized expectation propagation for graphical models over strings. In Proceedings of NAACL-HLT, pages 932– 942, Denver, June. Supplementary material (11 pages) also available. find
  6. Ryan Cotterell, Nanyun Peng, and Jason Eisner. 2014. Stochastic contextual edit distance and probabilistic FSTs. In Proceedings of ACL. find
  7. Mathias Creutz and Krista Lagus. 2005. Inducing the morphological lexicon of a natural language from unannotated text. In Proceedings of the International and Interdisciplinary Conference on Adaptive Knowledge Representation and Reasoning (AKRR05), volume 1. find
  8. Rina Dechter, Bozhena Bidyuk, Robert Mateescu, and Emma Rollon. 2010. On the power of belief propagation: A constraint propagation perspective. In Rina Dechter, Hector Geffner, and Joseph Y. Halpern, editors, Heuristics, Probability and Causality: A Tribute to Judea Pearl. College Publications. find
  9. Rina Dechter. 2003. Constraint Processing. Morgan Kaufmann. find
  10. Paul Boersma and Bruce Hayes. 2001. Empirical tests of the Gradual Learning Algorithm. Linguistic Inquiry, 32(1):45–86. find
  11. Markus Dreyer and Jason Eisner. 2009. Graphical models over multiple strings. In Proceedings of EMNLP, pages 101–110. find
  12. Paul Boersma. 1997. How we learn variation, optionality, and probability. In Proceedings of the Institute of Phonetic Sciences of the University of Amsterdam, volume 21, pages 43–58. find
  13. Markus Dreyer and Jason Eisner. 2011. Discovering morphological paradigms from plain text using a Dirichlet process mixture model. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 616–627, Edinburgh, July. Supplementary material (9 pages) also avail… find
  14. Paul Boersma. 1998. How we learn variation, optionality, and probability. In Functional Phonology: Formalizing the Interactions Between Articulatory and Perceptual Drives, chapter 15. Ph.D. Dissertation, University of Amsterdam. Previously appeared in IFA Proceedings (1997), pp. 43–58. find
  15. Alexandre Bouchard-Côté, Percy Liang, Thomas L. Griffiths, and Dan Klein. 2007. A probabilistic approach to language change. In Proceedings of NIPS. find
  16. Alexandre Bouchard-Côté, David Hall, Thomas L. Griffiths, and Dan Klein. 2013. Automated reconstruction of ancient languages using probabilistic models of sound change. Proceedings of the National Academy of Sciences. find
  17. Noam Chomsky and Morris Halle. 1968. The Sound Pattern of English. Harper and Row. find
  18. Christos Christodoulopoulos, Sharon Goldwater, and Mark Steedman. 2010. Two decades of unsupervised POS induction: How far have we come? In Proceedings of EMNLP, pages 575–584.
  19. Markus Dreyer, Jason R. Smith, and Jason Eisner. 2008. Latent-variable modeling of string transductions with finite-state methods. In Proceedings of EMNLP, pages 1080–1089. find
  20. Markus Dreyer. 2011. A Non-Parametric Model for the Discovery of Inflectional Paradigms from Plain Text Using Graphical Models over Strings. Ph.D. thesis, Johns Hopkins University, Baltimore, MD, April. find
  21. Sarah Eisenstat. 2009. Learning underlying forms with MaxEnt. Master’s thesis, Brown University, Providence, RI. find
  22. Jason Eisner. 2002a. Comprehension and compilation in Optimality Theory. In Proceedings of ACL, pages 56–63, Philadelphia, July. find
  23. Jason Eisner. 2002b. Discovering syntactic deep structure via Bayesian statistics. Cognitive Science, 26(3):255–268, May-June. find
  24. Jason Eisner. 2015. Should linguists evaluate grammars or grammar learners? In preparation. find
  25. C. Douglas Johnson. 1972. Formal Aspects of Phonological Description. Mouton. find
  26. Micha Elsner, Sharon Goldwater, and Jacob Eisenstein. 2012. Bootstrapping a unified model of lexical and phonetic acquisition. In Proceedings of ACL, pages 184–193. find
  27. René Kager. 1999. Optimality Theory, volume 2. MIT Press. find
  28. Micha Elsner, Sharon Goldwater, Naomi Feldman, and Frank Wood. 2013. A joint learning model of word segmentation, lexical acquisition, and phonetic variability. In Proceedings of EMNLP, pages 42–54. find
  29. Alexander M. Fraser, Marion Weller, Aoife Cahill, and Fabienne Cap. 2012. Modeling inflection and word-formation in SMT. In Proceedings of EACL, pages 664–674. find
  30. J. Goldsmith. 2006. An algorithm for the unsupervised learning of morphology. Natural Language Engineering, 12(4):353–371. find
  31. Sharon Goldwater and Mark Johnson. 2003. Learning OT constraint rankings using a maximum entropy model. In Jennifer Spenader, Anders Eriksson, and Osten Dahl, editors, Proceedings of the Workshop on Variation within Optimality Theory, pages 113–122, Stockholm University. find
  32. David Hall and Dan Klein. 2010. Finding cognate groups using phylogenies. In Proceedings of ACL. find
  33. Bruce Hayes and Colin Wilson. 2008. A maximum entropy model of phonotactics and phonotactic learning. Linguistic Inquiry, 39(3):379–440. find
  34. Matt Hohensee and Emily M. Bender. 2012. Getting more from morphology in multilingual dependency parsing. In Proceedings of NAACL-HLT, pages 315– 326. find
  35. Mans Hulden. 2009. Revisiting multi-tape automata for Semitic morphological analysis and generation. In Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages, pages 19–26, March.
  36. Sharon Inkelas and Cheryl Zoll. 2005. Reduplication: Doubling in Morphology. Number 106 in Cambridge Studies in Linguistics. Cambridge University Press. find
  37. Roman Jakobson. 1948. Russian conjugation. Word, 4:155–167. find
  38. Gaja Jarosz. 2006. Richness of the base and probabilistic unsupervised learning in Optimality Theory. In Proceedings of the Eighth Meeting of the ACL Special Interest Group on Computational Phonology and Morphology, pages 50–59. find
  39. Gaja Jarosz. 2013. Learning with hidden structure in optimality theory and harmonic grammar: Beyond robust interpretive parsing. Phonology, 30(01):27– 71. find
  40. Ronald M. Kaplan and Martin Kay. 1994. Regular models of phonological rule systems. Computational Linguistics, 20(3):331–378. find
  41. Michael J. Kenstowicz and Charles W. Kisseberth. 1979. Generative Phonology. Academic Press San Diego. find
  42. András Kornai. 1995. Formal Phonology. Garland Publishing, New York. find
  43. Zhifei Li and Jason Eisner. 2009. First- and second-order expectation semirings with applications to minimum-risk training on translation forests. In Proceedings of EMNLP, pages 40–51, Singapore, August.
  44. Dong C. Liu and Jorge Nocedal. 1989. On the limited memory BFGS method for large scale optimization. Mathematical Programming, 45(1-3):503–528. find
  45. Andrew McCallum, Dayne Freitag, and Fernando C. N. Pereira. 2000. Maximum entropy Markov models for information extraction and segmentation. In Proceedings of ICML, pages 591–598. find
  46. Navarré Merchant. 2008. Discovering Underlying Forms: Contrast Pairs and Ranking. Ph.D. thesis, Rutgers University. Available on the Rutgers Optimality Archive as ROA-964. find
  47. Kevin P. Murphy, Yair Weiss, and Michael I. Jordan. 1999. Loopy belief propagation for approximate inference: An empirical study. In Proceedings of UAI, pages 467–475. find
  48. Joe Pater, Karen Jesney, Robert Staubs, and Brian Smith. 2012. Learning probabilities over underlying representations. In Proceedings of the Twelfth Meeting of the Special Interest Group on Computational Morphology and Phonology, pages 62–71. find
  49. Judea Pearl. 1988. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. find
  50. Nanyun Peng, Ryan Cotterell, and Jason Eisner. 2015. Dual decomposition inference for graphical models over strings. In Proceedings of EMNLP, Lisbon, September. To appear. find
  51. Janet Pierrehumbert. 2003. Probabilistic phonology: Discrimination and robustness. In Probabilistic Linguistics, pages 177–228. MIT Press. find
  52. Alan Prince and Paul Smolensky. 2004. Optimality Theory: Constraint Interaction in Generative Grammar. Wiley-Blackwell. find
  53. Jason A. Riggle. 2004. Generation, Recognition, and Learning in Finite State Optimality Theory. Ph.D. thesis, University of California at Los Angeles. find
  54. Jason Riggle. 2012. Phonological features chart (version 12.12). Available online at https://dl.dropboxusercontent.com/u/ 5956329/Riggle/PhonChart_v1212.pdf (retrieved 2015-08-07). ↗
  55. David Sankoff. 1978. Probability and linguistic variation. Synthese, 37(2):217–238. find
  56. Ferdinand de Saussure. 1916. Course in General Linguistics. Columbia University Press. English edition of June 2011, based on the 1959 translation by Wade Baskin. find
  57. Paul Smolensky and Géraldine Legendre. 2006. The Harmonic Mind: From Neural Computation to Optimality-Theoretic Grammar (Vol. 1: Cognitive Architecture). MIT Press. find
  58. Bruce Tesar and Paul Smolensky. 1998. Learnability in Optimality theory. Linguistic Inquiry, 29(2):229– 268. find
  59. Bruce Tesar. 2004. Contrast analysis in phonological learning. Technical Report ROA-695, Rutgers Optimality Archive. find
  60. Bruce Tesar. 2014. Output-Driven Phonology: Theory and Learning. Cambridge University Press. find
  61. Yi Yang and Jacob Eisenstein. 2013. A log-linear model for unsupervised text normalization. In EMNLP, pages 61–72. find
  62. Reyyan Yeniterzi. 2011. Exploiting morphology in Turkish named entity recognition system. In Proceedings of the ACL Student Session, pages 105– 110. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cotterell-2015-modeling-word-forms-latent-underlying/