Elamur Research⌕
☰
home / papers / 2022 / HeiMorph at SIGMORPHON 2022 Shared Task on Morphological Acquisition Trajectories

HeiMorph at SIGMORPHON 2022 Shared Task on Morphological Acquisition Trajectories

Paper · 2022 · Akhilesh Kakolu Ramarao, Yulia Zinova, Kevin Tang et al.

Summary

The HeiMorph team took part in Part 2 of SIGMORPHON 2022 Task 0. The task asks a model to inflect words from very small training sets and checks whether its learning curve looks like child language acquisition. The data covered three patterns: English past tense, German noun plurals and Arabic noun plurals, with between 100 and 1,000 training examples each. The authors trained character-level Transformers in Fairseq. To make up for the scarce data, they added synthetic examples. Their baseline augmentation followed Anastasopoulos and Neubig (2019) and swaps aligned stem characters for random letters. They also tested a variant that keeps a substitute letter only if that letter followed the previous character somewhere in the training data. This keeps fake stems phonotactically plausible. Their submitted systems beat the neural baseline in all three languages. The bigram-aware variant improved results in just two languages, English and Arabic, and even there the benefit appeared solely in settings with a small number of synthetic examples. Across all conditions, the models showed no U-shaped acquisition curve. The work suggests that augmentation choices depend heavily on the language and the amount added, so they need careful inspection.

Key points

  • Average test accuracy was 55.5% for Arabic, 67% for German and 73.38% for English. These are gains of 2.66, 8.69 and 4.5 points over the neural baseline.
  • The model setup:
    • Fairseq Transformer with 4 encoder and 4 decoder layers, 4 heads, embedding size 256 and hidden size 1,024.
    • Adam optimizer with a learning rate of 0.001 and batch size 400, at most 3,000 updates.
    • Beam width 5 at decoding.
    • The model input is the lemma's characters plus the source and target tags, separated by #. For example, t a k e <V;PRS> # <V;PST> maps to t o o k.
  • Each training size was combined with 1,000 or 10,000 hallucinated examples, using either unigram (random) or bigram-constrained substitution.
  • The submission policy was picked from German results alone. Sizes below 500 got 10,000 hallucinations and larger sizes got 1,000. In hindsight, bigram augmentation with 1,000 hallucinations would have been the better choice for English and Arabic.
  • Results are reported as both accuracy and Levenshtein distance per training size. For example, English test accuracy rose from 65.2% at 100 examples to 80% at 1,000.

Interesting ideas

  • I1. When creating synthetic stems, the authors accept a random replacement letter only if the resulting character bigram was seen in real data, so fake words stay pronounceable.
  • I2. The amount of augmentation behaved like a sensitive hyperparameter: 1,000 synthetic examples sometimes beat 10,000.
  • I3. The team tuned on one language and assumed the result would carry over, which turned out to be wrong; this is a useful cautionary example for evaluation design.
  • I4. Reporting edit distance next to exact-match accuracy shows how close the wrong outputs are, not just how often they fail.
  • I5. Accuracy is tracked across a ladder of training sizes to look for acquisition-like patterns such as over-regularization of irregular forms.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Synthetic lemma–form pairs for training English inflection or lemmatization rules could pass the same bigram filter, so the analyzer never learns from impossible letter strings.
  • I2 (learn): The share of silver or synthetic data in training should be swept and evaluated, not maximized, with a no-augmentation run kept as a control.
  • I3 (learn): Settings chosen on one genre or treebank should be rechecked on each target condition before adoption, and each decision should be logged with the slice it was based on.
  • I4 (tokens): A lemmatizer or inflection generator could report mean Levenshtein distance on its errors, which separates near-misses (e.g., spelling of a suffix) from wholesale wrong forms.
  • I5 (learn): Mova could measure irregular-verb accuracy across checkpoints or data sizes, to detect whether self-induced repair rules over-regularize before stabilizing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Heinrich Heine University DüsseldorfDüsseldorf · Germany33 works · 1,487 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksMorphology · 405 works

Cite

Cite as Akhilesh Kakolu Ramarao, Yulia Zinova, Kevin Tang and Ruben van de Vijver. 2022. HeiMorph at SIGMORPHON 2022 Shared Task on Morphological Acquisition Trajectories. Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.sigmorphon-1.24

@inproceedings{ramarao2022heimorph,
  title = {{HeiMorph at SIGMORPHON 2022 Shared Task on Morphological Acquisition Trajectories}},
  author = {Akhilesh Kakolu Ramarao, Yulia Zinova, Kevin Tang and Ruben {van de Vijver}},
  booktitle = {Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology},
  year = {2022},
  doi = {10.18653/v1/2022.sigmorphon-1.24},
  url = {https://doi.org/10.18653/v1/2022.sigmorphon-1.24}
}

References

23 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fauzia Abdalla, Khawla Aljenaie, Abdessatar Mah-foudhi, Edith L Bavin, and Letitia R Naigles. 2012. Plural noun inflection in kuwaiti arabic-speaking children with and without specific language impairment*. Journal of child language, 40(1):139–168. ↗
  2. Farrell Ackerman and Robert Malouf. 2013. Morphological organization: The low conditional entropy conjecture. Language, 89(3):429–464. ↗
  3. Antonios Anastasopoulos and Graham Neubig. 2019. Pushing the limits of low-resource morphological inflection. arXiv preprint arXiv:1908.05838. ↗
  4. Elabbas Benmamoun, Abdulkafi Albirini, Silvina A. Montrul, and Eman Saadah. 2014. Arabic plurals and root and pattern morphology in palestinian and egyptian heritage speakers. Linguistic Approaches to Bilingualism, 4(1):89–123. ↗
  5. Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bansal, and Diyi Yang. 2021. An empirical survey of data augmentation for limited data learning in nlp. ↗
  6. Harald Clahsen, Monika Rothweiler, Andreas Woest, and Gary F Marcus. 1992. Regular and irregular inflection in the acquisition of german noun plurals. Cognition, 45(3):225–255. ↗
  7. Lisa Garnand Dawdy-Hesterberg and Janet Breckenridge Pierrehumbert. 2014. Learnability and generalisation of arabic broken plural nouns. Language, cognition and neuroscience, 29(10):1268–1282. ↗
  8. Naranjo Matías Guzmán. 2020. Analogy, complexity and predictability in the russian nominal inflection system. Morphology, pages 1–44. ↗
  9. Martin Haspelmath and Andrea Sims. 2013. Understanding morphology. Routledge. find
  10. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. ↗
  11. Christo Kirov, Ryan Cotterell, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sabrina J Mielke, Arya D McCarthy, Sandra Kübler, et al. 2018. Unimorph 2.0: universal morphology. arXiv preprint arXiv:1810.11101. ↗
  12. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  13. Michael Ramscar, Melody Dye, and Stewart M Mc-Cauley. 2013. Error and expectation in language learning: The curious absence of mouses in adult speech. Language, pages 760–793. ↗
  14. Dorit Ravid and Rola Farah. 1999. Learning about noun plurals in early palestinian arabic. First Language, 19(56):187–206. ↗
  15. Elinor Saiegh-Haddad, Areen Hadieh, and Dorit Ravid. 2012. Acquiring noun plurals in palestinian arabic: Morphology, familiarity, and pattern frequency. Language Learning, 62(4):1079–1109. ↗
  16. Farhan Samir and Miikka Silfverberg. 2022. One wug, two wug+ s transformer inflection models hallucinate affixes. In Proceedings of the Fifth Workshop on the Use of Computational Methods in the Study of Endangered Languages, pages 31–40. ↗
  17. Anne-Michelle Tessier. 2019. U-shaped development in error-driven child phonology. Wiley Interdisciplinary Reviews: Cognitive Science, 10(6):e1505. ↗
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.
  19. Shijie Wu, Ryan Cotterell, and Mans Hulden. 2020. Applying the transformer to character-level transduction. ↗
  20. Jordan Kodner and Salam Khalifa. 2022. SIGMORPHON-UniMorph 2022 Shared Task 0: Modeling Inflection in Language Acquisition. In Proceedings of the SIGMORPHON 2022 Shared Task: Morphological Inflection, Seattle. North American Chapter of the Association for Computational Linguistics. find
  21. Robert Malouf. 2017. Abstractive morphological learning with a recurrent neural network. Morphology, 27(4):431–458. find
  22. Gary F Marcus, Ursula Brinkmann, Harald Clahsen, Richard Wiese, and Steven Pinker. 1995. German inflection: The exception that proves the rule. Cognitive psychology, 29(3):189–256. ↗
  23. Gary F Marcus, Steven Pinker, Michael Ullman, Michelle Hollander, T John Rosen, Fei Xu, and Harald Clahsen. 1992. Overregularization in language acquisition. In Monographs of the society for research in child development. JSTOR. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ramarao-2022-heimorph-sigmorphon-2022-shared-task/