Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages
Paper · 2019 · Shauli Ravfogel, Yoav Goldberg and Tal Linzen
Summary
Comparing how well RNNs learn syntax across languages is hard: real languages differ in several typological properties simultaneously, and models get trained on different corpora. The authors isolate individual properties by building synthetic variants of English from one expert-annotated source, the English Penn Treebank converted to Universal Dependencies with the Stanford converter. They rewrite the dependency trees to change one parameter at a time: constituent order of subject, object and verb, the presence of polypersonal (subject plus object) verb agreement, and the type of overt case marking. A bidirectional LSTM then sees a sentence with one verb hidden and predicts whether that verb's subject and object are singular or plural. Three main findings emerged. SVO order gave better results than SOV, which suggests the networks favor nearby dependencies. Predicting subject and object together outperformed predicting each alone. Explicit case suffixes made the task much easier under every word order tested. The paradigm offers a controlled way to study how architectural inductive biases interact with typology. The generation code is publicly released.
Key points
- Model: a BiLSTM with 150 hidden units feeding separate MLPs (hidden layers of 100 and 50) for subject and object plurality. Word embeddings are summed with character n-gram embeddings (lengths 1–5). Each result averages four random initializations. The training corpus has about 35,000 treebank sentences.
- Polypersonal agreement: joint prediction beat single-task prediction for subjects (95.7% vs 94.7%), for objects (90.0% vs 88.9%) and for object recall (85.4% vs 81.8%).
- Case marking: with unambiguous case suffixes (e.g., -kar, -kon, -ker), accuracy exceeded 98% for VOS, OVS and flexible word order.
- Under flexible word order, even marking only argument plurality, with no grammatical-function distinction, raised accuracy to 96.0% for subjects and 86.1% for objects. Without case, the figures were 88.6% and 60.2%.
- Code for generating the synthetic languages is available at github.com/Shaul1321/rnn_typology.
Fields, topics and concepts
📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works
Cite
Cite as Shauli Ravfogel, Yoav Goldberg and Tal Linzen. 2019. Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
@inproceedings{ravfogel2019studying,
title = {{Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages}},
author = {Shauli Ravfogel, Yoav Goldberg and Tal Linzen},
booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
year = {2019},
doi = {10.18653/v1/n19-1356},
url = {https://doi.org/10.18653/v1/n19-1356}
}References
- \#\# Conclusions Matthew Baerman and Dunstan Brown. 2013. Case syncretism. In Matthew S. Dryer and Martin Haspelmath, editors, The World Atlas of Language Structures Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/ravfogel-2019-studying-inductive-biases-rnns-synthetic/