Elamur Research⌕
☰
home / papers / 2020 / The Importance of Category Labels in Grammar Induction with Child-directed Utterances

The Importance of Category Labels in Grammar Induction with Child-directed Utterances

Summary

This paper concerns the evaluation of grammars that a model induces from raw text without supervision. Such grammars are usually compared only by their bracketing structure (unlabeled F1). Phrasal category labels are ignored, even though they are an essential part of a grammar. The authors propose the Recall-Homogeneity (RH) metric. It multiplies the unlabeled constituent recall by the homogeneity of labels in the constituents that match the gold standard. Unlike RVM, RH does not penalize a model for categories that are finer-grained than the gold ones. The inducer is a Bayesian Dirichlet-multinomial PCFG model in Chomsky normal form with Gibbs sampling. The data are adult child-directed speech from the CHILDES corpus: English (Adam, Eve), Mandarin (Tong) and German (Leo). WSJ20Dev is also used for comparison. Under unlabeled evaluation, dense, almost purely right-branching grammars appear to be the best. RH, by contrast, confirms the linguistically expected advantage of sparse grammars and of a larger number of categories. Bounding center-embedding depth remains beneficial under labeled evaluation as well. Hence, conclusions about grammar induction that do not take labels into account may be wrong.

Key points

  • Data: for each language, over 15,000 caregiver utterances from CHILDES. The English trees for Adam and Eve are hand-annotated. The gold trees for Tong and Leo were produced automatically by the Kitaev and Klein (2018) parser. Each model was run 10 times for 700 iterations.
  • Parameter β (Dirichlet concentration): unlabeled F1 peaks at high β, whereas RH peaks at low β, which is consistent with the sparsity of human grammars. As an example, the authors cite the Penn Treebank: 73 nonterminal categories and only 17,020 attested rules out of over 28 million possible ones. On WSJ20Dev the trend is the same but weaker.
  • Number of categories C: at β = 0.01, RH increases with C and peaks at C = 75. For comparison, Adam has 63 unique tags, while previous work used 30 categories or fewer.
  • Depth bounding: on Adam, the best RH is achieved with depth 3, which is statistically significantly better than the unbounded model (p < 10⁻³). The result was confirmed on Eve, Tong and Leo: the difference in unlabeled F1 is significant at p < 0.001. In these experiments C = 30 due to hardware limitations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Psycholinguistics & language acquisition

Cite

Cite as Lifeng Jin and William Schuler. 2020. The Importance of Category Labels in Grammar Induction with Child-directed Utterances. Proceedings of the 16th International Conference on Parsing Technologies and the IWPT 2020 Shared Task on Parsing into Enhanced Universal Dependencies.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.iwpt-1.15 · arXiv 2006.11646

@inproceedings{jin2020importance,
  title = {{The Importance of Category Labels in Grammar Induction with Child-directed Utterances}},
  author = {Lifeng Jin and William Schuler},
  booktitle = {Proceedings of the 16th International Conference on Parsing Technologies and the IWPT 2020 Shared Task on Parsing into Enhanced Universal Dependencies},
  year = {2020},
  doi = {10.18653/v1/2020.iwpt-1.15},
  eprint = {2006.11646},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.iwpt-1.15}
}

References

30 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. João V. Graça, Kuzman Ganchev, Taskar Ben, and Fernando Pereira. 2009. Posterior vs. Parameter sparsity in latent variable models. In Advances in Neural Information Processing Systems, pages 664–672. find
  2. Omri Abend, Tom Kwiatkowski, Nathaniel J. Smith, Sharon Goldwater, and Mark Steedman. 2017. Bootstrapping language acquisition. In Cognition, volume 164, pages 116–143. Elsevier B.V. find
  3. Lifeng Jin, Finale Doshi-Velez, Timothy Miller, Lane Schwartz, and William Schuler. 2019. Unsupervised Learning of PCFGs with Normalizing Flow. In ACL. find
  4. Colin Bannard, Elena Lieven, and Michael Tomasello. 2009. Modeling children’s early grammatical knowledge. Proceedings of the National Academy of Sciences of the United States of America, 106(41):17284–9. find
  5. Lifeng Jin, Finale Doshi-Velez, Timothy A Miller, William Schuler, and Lane Schwartz. 2018a. Depth-bounding is effective: Improvements and evaluation of unsupervised PCFG induction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  6. Heike Behrens. 2006. The input–output relationship in first language acquisition. Language and Cognitive Processes, 21(1-3):2–24. find
  7. Roger Brown. 1973. A first language: The early stages. Harvard U. Press. find
  8. Noam Chomsky. 1965. Aspects of the Theory of Syntax. MIT Press, Cambridge, MA. different runs are not exchangeable. find
  9. Lifeng Jin, Finale Doshi-Velez, Timothy A Miller, William Schuler, and Lane Schwartz. 2018b. Unsupervised Grammar Induction with Depth-bounded PCFG. Transactions of the Association for Computational Linguistics. our card
  10. Lifeng Jin and William Schuler. 2019. Variance of average surprisal: a better predictor for quality of grammar from unsupervised PCFG induction. In ACL. find
  11. Mark Johnson, Thomas L. Griffiths, and Sharon Goldwater. 2007. Bayesian Inference for PCFGs via Markov chain Monte Carlo. Proceedings of Human Language Technologies: The Conference of the North American Chapter of the Association for Computational Linguistics, pages 139–146. our card
  12. Yoav Seginer. 2007. Fast Unsupervised Incremental Parsing. In Proceedings of the Annual Meeting of the Association of Computational Linguistics, pages 384–391. find
  13. Nenaugh Kemp, Elena Lieven, and Michael Tomasello. 2005. Young Children’s Knowledge of the “Determiner” and “Adjective” Categories. Journal of Speech, Language, and Hearing Research, 48(June):592–609. find
  14. Cory Shain, William Bryce, Lifeng Jin, Victoria Krakovna, Finale Doshi-Velez, Timothy Miller, William Schuler, and Lane Schwartz. 2016. Memory-bounded left-corner unsupervised grammar induction on child-directed input. In Proceedings of the International Conference on Computational Linguistics, page… find
  15. Yoon Kim, Chris Dyer, and Alexander M Rush. 2019. Compound Probabilistic Context-Free Grammars for Grammar Induction. In ACL. our card
  16. Yikang Shen, Zhouhan Lin, Chin-Wei Huang, and Aaron Courville. 2018. Neural Language Modeling by Jointly Learning Syntax and Lexicon. In ICLR. find
  17. Nikita Kitaev and Dan Klein. 2018. Constituency Parsing with a Self-Attentive Encoder. In ACL. our card
  18. Tom Kwiatkowski, Sharon Goldwater, Luke Zettlemoyer, and Mark Steedman. 2012. A probabilistic model of syntactic and semantic acquisition from child-directed utterances and their meanings. Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pa… our card
  19. Brian Macwhinney. 1992. The CHILDES Project: Tools for Analyzing Talk, third edition. Lawrence Elrbaum Associates, Mahwah, NJ. find
  20. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. our card
  21. Amy Perfors, Joshua B Tenenbaum, and Terry Regier. 2011. The learnability of abstract syntactic principles. Cognition, 118:306–338. find
  22. Julian M. Pine, Daniel Freudenthal, Grzegorz Krajew-ski, and Fernand Gobet. 2013. Do young children have adult-like syntactic categories? Zipf’s law and the case of the determiner. Cognition, 127(3):345– 360. find
  23. Elias Ponvert, Jason Baldridge, and Katrin Erk. 2011. Simple unsupervised grammar induction from raw text with cascaded finite state models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1077–1086. find
  24. Andrew Rosenberg and Julia Hirschberg. 2007. V-measure: A conditional entropy-based external cluster evaluation measure. In Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL). our card
  25. William Schuler, Samir AbdelRahman, Tim Miller, and Lane Schwartz. 2010. Broad-coverage parsing using human-Like memory constraints. Computational Linguistics, 36(1):1–30. find
  26. Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2019. Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks. In ICLR. find
  27. Haoyue Shi, Jiayuan Mao, Kevin Gimpel, and Karen Livescu. 2019. Visually Grounded Neural Syntax Acquisition. In ACL. find
  28. Mark Steedman. 2002. Formalizing Affordance. In Proceedings of the Annual Meeting of the Cognitive Science Society. find
  29. Michael Tomasello and Raquel Olguin. 1993. Twenty-three-month-old children have a grammatical category of noun. Cognitive Development, 8(4):451– 464. find
  30. Virginia Valian. 1986. Syntactic Categories in the Speech of Young Children. Developmental Psychology, 22(4):562–579. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jin-2020-importance-category-labels-grammar-induction/