Elamur Research⌕
☰
home / papers / 2022 / Learning English with Peppa Pig

Learning English with Peppa Pig

Paper · 2022 · Mitja Nikolaus, Afra Alishahi and Grzegorz Chrupała

Summary

Neural models that learn spoken language by linking it to visual input are usually trained on images or videos paired with spoken descriptions. This makes speech and vision correlate far more tightly than in a child's actual experience. The authors test whether learning still works when that link is weak. Their data is the children's cartoon Peppa Pig: 209 annotated episodes, split into character dialog and narrator comments. Dialog often refers to things not on screen, such as plans or past events, and it is mixed up with speaker voices and background sounds. A simple bi-modal model with separate audio and video encoders, mapped into a shared space, is trained only on dialog. It is then tested on narration, which is descriptive and comes from one voice never heard in training. Evaluation uses clip retrieval and controlled minimal-pair tests modelled on infant looking experiments. The model does learn visual meaning for whole utterances and for frequent nouns and verbs. Self-supervised audio pre-training followed by fine-tuning matters most. The result shows that grounded learning can survive noisy, realistic input.

Key points

  • Data comes from Papasarantopoulos and Cohen (2021): seasons 1–5 with aligned transcripts and dialog/narration labels. Transcripts were re-aligned with gentle where word-level timing was needed.
  • Splits:
    • Train: dialog only, about 10 h and 15,666 clips.
    • Validation: dialog (0.66 h, 1,026 clips) and narration (0.94 h, 1,467 clips).
    • Test: narration only (0.64 h, 1,006 clips).
  • Ablations covered encoder pre-training, temporal information and segmentation. One segmentation scheme ("jitter") varies the audio and video lengths independently around a 2.3 s mean.
  • Temporal information helps on longer segments but not on very short ones. Minimal-pair clips average 0.44 s, and shuffling their frames caused no significant drop.
  • Per-word accuracy varies widely:
    • Weakest: house, car, cake, which have varied referents or appear in displaced speech.
    • Strongest: ice, box, sand.

Interesting ideas

  • I1. Train on the messy, conversational part of the data and test on a cleaner, descriptive part with an unseen speaker, so that success cannot come from shortcuts tied to the source.
  • I2. Minimal-pair tests adapted from infant preferential-looking studies offer two candidates that differ in exactly one concept and check which one the model prefers.
  • I3. Shuffling video frames acts as a negative control that reveals whether temporal order is really being used.
  • I4. The error analysis tied poorly learned words to displaced talk about absent things and to referents that shift (whole cake vs. slice vs. batter).

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver data and repair rules could be induced from dialog-heavy texts and scored on held-out narrative fables, to check that gains carry across genres.
  • I2 (answer): Readers could get paired world states that differ in one entity or action, and must pick the one that matches the sentence, which isolates lexical understanding.
  • I3 (learn): Scrambling sentence or event order before world-building would show which answers truly depend on event sequencing and which only appear to.
  • I4 (frames): Utterances about plans, past events or hypotheticals could be marked as displaced, so they become irrealis states instead of facts in the current world. Polysemous part/substance nouns could also be flagged for the ontology.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tilburg UniversityTilburg · Netherlands50 works · 6,429 citations🎓Aix-Marseille UniversitéMarseille · France24 works · 39,160 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Mitja Nikolaus, Afra Alishahi and Grzegorz Chrupała. 2022. Learning English with Peppa Pig. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00498

@article{nikolaus2022learning,
  title = {{Learning English with Peppa Pig}},
  author = {Mitja Nikolaus, Afra Alishahi and Grzegorz Chrupała},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.1162/tacl\_a\_00498},
  url = {https://doi.org/10.1162/tacl_a_00498}
}

References

44 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Afra Alishahi, Marie Barking, and Grzegorz Chrupała. 2017. Encoding of phonology in a recurrent neural model of grounded speech. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 368–378, Vancouver, Canada, Association for Computational Linguistics. https://doi.org/10.18653/v1/K17 -1037 ↗
  2. Grzegorz Chrupała. 2022. Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques. Journal of Artificial Intelligence Research, 73:673–707. https://doi.org/10 .1613/jair.1.12967 ↗
  3. Daniel R. Anderson and Tiffany A. Pempek. 2005. Television and very young children. American Behavioral Scientist, 48(5):505–522. https://doi.org/10.1177/00027642042 71506 ↗
  4. Grzegorz Chrupała, Lieke Gelderloos, and Afra Alishahi. 2017. Representations of language in a model of visually grounded speech signal. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 613–622, Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. Wav2vec Vancouver, Canada, Association for Computational Linguistics. https://doi.org/10 .18653/v1/P17-1057 ↗
  5. Glass. 2018. Jointly discovering visual objects and spoken words from raw sensory input. In Proceedings of the European Conference on Computer Vision (ECCV), pages 649–665. https://doi.org/10.1007/978-3-030-01231-1 40 ↗
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  7. David F. Harwath, Antonio Torralba, and James R. Glass. 2016. Unsupervised learning of spoken language with visual context. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 1858–1866. find
  8. Michael C. Frank, Mika Braginsky, Daniel Yurovsky, and Virginia A. Marchman. 2021. Variability and Consistency in Early Language Learning: The Wordbank Project, MIT Press. https://doi.org/10.7551/mitpress/11577.001.0001 ↗
  9. William N. Havard, Jean-Pierre Chevrot, and Laurent Besacier. 2019a. Models of visually grounded speech signal pay attention to nouns: A bilingual experiment on English and Japanese. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019, pages 8618–8622. IEEE. https://doi.org/10.1109/ICASSP.2019 .8683069 ↗
  10. Peter Gorniak and Deb Roy. 2003. A visually grounded natural language interface for reference to spatial scenes. In Proceedings of the 5th International Conference on Multimodal Interfaces, pages 219–226. https://doi.org/10.1145/958432.958474 ↗
  11. William N. Havard, Jean-Pierre Chevrot, and Laurent Besacier. 2019b. Word recognition, competition, and activation in a model of visually grounded speech. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 339–348, Hong Kong, China. Association for Computational Linguistics. https://doi.org/10 .18653/v1/K19-1032 ↗
  12. David Harwath and James Glass. 2015. Deep multimodal semantic embeddings for speech and images. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pages 237–244. IEEE. https://doi.org/10.1109/ASRU.2015.7404800 ↗
  13. David Harwath and James Glass. 2017. Learning word-like units from joint audio-visual analysis. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 506–517, Vancouver, Canada. Association for Computational Linguistics. https://doi.org/10.18653/v1/P17-1047 ↗
  14. Kathy Hirsh-Pasek and Roberta Michnick Golinkoff. 1996. The intermodal preferential looking paradigm: A window onto emerging language comprehension. In D. McDaniel, C. McKee, and H. S. Cairns, editors, Methods for assessing children’s syntax. The MIT Press. ↗
  15. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460. ↗
  16. David Harwath and James R. Glass. 2019. Towards visually grounded sub-word speech unit discovery. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019, pages 3017–3021. IEEE. https://doi.org/10.1109/ICASSP.2019 .8682666 ↗
  17. Wei-Ning Hsu, David Harwath, and James Glass. 2019. Transfer learning from audio-visual grounding to speech recognition. In Proceedings of Interspeech 2019, pages 3242–3246. David Harwath, Adria Recasens, Dı́dac Surı́s, Galen Chuang, Antonio Torralba, and James https://doi.org/10.21437/Interspeech .2019-1227 ↗
  18. Andrej Karpathy, Armand Joulin, and Fei-Fei Li. 2014. Deep fragment embeddings for bidirectional image sentence mapping. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 1889–1897. ↗
  19. Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, TrevorBack, PaulNatsev, Mustafa Suleyman, and Andrew Zisserman. 2017. The Kinetics human action video dataset. CoRR, abs/1705.06950. ↗
  20. Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 2630–2640. IEEE. https://doi.org/10.1109/ICCV.2019.00272 ↗
  21. Mathew Monfort, SouYoung Jin, Alexander Liu, David Harwath, Rogerio Feris, James Glass, and Aude Oliva. 2021. Spoken moments: Learning joint audio-visual representations from video descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/CVPR46437.2021.01463 ↗
  22. Khazar Khorrami and Okko Räsänen. 2021. Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? A computational investigation. Language Development Research, 1:123–191. https://doi.org/10.31234/osf.io/37zna ↗
  23. Niloy Mukherjee and Deb Roy. 2003. A visual context-aware multimodal system for spoken language processing. In Eighth European Conference on Speech Communication and Technology. find
  24. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. ↗
  25. Mitja Nikolaus and Abdellah Fourtassi. 2021. Evaluating the acquisition of semantic knowledge from cross-situational learning in artificial neural networks. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 200–210, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.cmcl-1.24 ↗
  26. Natassa Kokla. 2021. Peppa Pig: An innovative way to promote formulaic language in pre-primary EFL classrooms. Research Papers in Language Teaching & Learning, 11(1). find
  27. Patricia K. Kuhl, Feng-Ming Tsao, and Huei-Mei Liu. 2003. Foreign-language experience in infancy: Effects of short-term exposure and social interaction on phonetic learning. Proceedings of the National Academy of Sciences, 100(15):9096–9101. https://doi.org/10.1073/pnas.1532872100 ↗
  28. Claire H. Noble, Caroline F. Rowland, and Julian M. Pine. 2011. Comprehension of argument structure and semantic roles: Evidence from English-learning children and the forced-choice pointing paradigm. Cognitive Science, 35(5):963–982. https://doi.org/10.1111/j.1551-6709.2011.01175.x Yevgen Matusevych, Afra Alishahi, and Paul Vogt. 2013. Automatic generation of naturalistic child-adult interaction data. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 35. ↗
  29. Nikos Papasarantopoulos and Shay B. Cohen. 2021. Narration generation for cartoon videos. Preprint: https://arxiv.org/abs/2101 .06803v1 ↗
  30. Danny Merkx, Stefan L. Frank, and Mirjam Ernestus. 2019. Language learning using speech to image retrieval. In Proceedings of Interspeech 2019, pages 1841–1845. https://doi.org/10.21437/Interspeech.2019 -3067 Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc. https://doi.org/10.1016/S0885 -2308(02)00024-4 ↗
  31. Deb K. Roy and Alex P. Pentland. 2002. Learning words from sights and sounds: A computational model. Cognitive Science, 26(1):113–146. https://doi.org/10.1207/s155167 09cog2601 4 ↗
  32. Jenny R. Saffran, Richard N. Aslin, and Elissa L. Newport. 1996. Statistical learning by 8-month-old infants. Science, 274(5294): 1926–1928. https://doi.org/10.1126/science.274.5294.1926 ↗
  33. Puyuan Peng and David Harwath. 2022. Fast-slow transformer for visually grounding speech. In Proceedings of the 2022 International Conference on Acoustics, Speech and Signal Processing. https://doi.org/10.1109/ICASSP43922.2022.9747103 ↗
  34. Thomas Schatz. 2016. ABX-Discriminability Measures and Applications. Ph.D. thesis, Université Paris 6 (UPMC). ↗
  35. Michael B. Robb, Rebekah A. Richert, and Ellen A. Wartella. 2009. Just a talking book? Word learning from watching baby videos. British Journal of Developmental Psychology, 27(1):27–45. https://doi.org/10 .1348/026151008X320156 ↗
  36. Paweł Scheffler, Christian Jones, and Anna Domińska. 2021. The Peppa Pig television series as input in pre-primary EFL instruction: A corpus-based study. International Journal of Applied Linguistics, 31(1):3–17. https://doi.org/10.1111/ijal.12298 ↗
  37. Andrew Rouditchenko, Angie Boggust, David Harwath, Brian Chen, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Hilde Kuehne, Rameswar Panda, Rogerio Feris, Brian Kingsbury, Michael Picheny, Antonio Torralba, and James Glass. 2021. AVLnet: Learning audio-visual language representations from instructional videos. In Proceedings of Interspeech 2021, pages 1584–1588. https://doi.org/10.21437/Interspeech.2021 -1312 ↗
  38. Gabriel Synnaeve, Maarten Versteegh, and Emmanuel Dupoux. 2014. Learning words from images and speech. In NIPS Workshop on Learning Semantics. ↗
  39. Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. 2018. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450–6459. https://doi.org/10 .1109/CVPR.2018.00675 ↗
  40. Brandon C. Roy, Michael C. Frank, Philip DeCamp, Matthew Miller, and Deb Roy. 2015. Predicting the birth of a spoken word. Proceedings of the National Academy of Sciences, 112(41):12663–12668. https://doi.org/10.1073/pnas.1419773112 ↗
  41. Robert Vanderplank. 2010. Déjà vu? A decade of research on language laboratories, television and video in language learning. Language Teaching, 43(1):1–37. https://doi.org/10.1017/S0261444809990267 ↗
  42. Deb Roy. 1999. Learning from Sights and Sounds: A Computational Model. Ph.D. thesis, MIT Media Laboratory. find
  43. Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. 2014. Learning deep features for scene recognition using Places database. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc. find
  44. Deb K. Roy. 2002. Learning visually grounded words and syntax for a scene description task. Computer Speech & Language, 16(3-4):353–385. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nikolaus-2022-learning-english-peppa-pig/