Elamur Research⌕
☰
home / papers / 2023 / Towards more Human-like Language Models based on Contextualizer Pretraining Strategy

Towards more Human-like Language Models based on Contextualizer Pretraining Strategy

Summary

The paper asks how to pretrain a capable masked language model when data is as scarce as what a child hears, within the BabyLM Challenge. The authors argue that always presenting a passage next to text from the same domain lets the model rely on spurious co-occurrences rather than real token dependencies, a problem they call a "contextualization trap." Their remedy, Contextualizer, is a data-augmentation scheme. An initial pass uses chunks built in a fixed easy-to-hard dataset order. After that, the original inputs are repeatedly reshuffled and repacked into fixed-length chunks, so each text is cut at new positions and placed beside different, often unrelated neighbours. A RoBERTa-base model trained from scratch this way on the 100M-word track comes within 1.2% of RoBERTa and performs on par with BERT on BLiMP. It uses about 1/300 of RoBERTa's pretraining data and needs a single GPU for roughly four days. The work suggests that varying context, rather than adding data, can substantially improve sample efficiency in compute-limited pretraining.

Key points

  • Data and curriculum: The study uses two data budgets, 10M and 100M tokens, from the BabyLM tracks. These are about 1/3000 and 1/300 of RoBERTa's training data. The ten sources run from easiest to hardest: CHILDES comes first, followed by Switchboard and OpenSubtitles; next are BNC, QED and CBT, then Children Stories and Gutenberg, with Simple Wikipedia and Wikipedia last. This order was checked with readability and age-of-acquisition statistics from the TCT toolkit.
  • Padding variants: Four schemes are compared: Clean Padding, Noisy Padding, Context-fixed Padding, and Context-augmented Padding. The last one reshuffles and re-packs inputs each time and gives the largest gains. The best recipe applies it 40 times.
  • Setup:
    • RoBERTa-base architecture with random initialization.
    • BPE tokenizer with a 50k vocabulary; 10k and 30k vocabularies hurt most BLiMP tasks.
    • Chunk length of 64 for the 10M track and 128 for the 100M track.
    • Training on one RTX 3090: about 6–8 hours for 10M and 3–4 days for 100M.
  • Released models: Contextualizer-RoBERTa-base-10M-v1 and Contextualizer-RoBERTa-base-100M, submitted to the Dynabench leaderboard.
  • Memorization probe: A model that saw identical chunks 40 times was compared with one that saw 40 different re-packings. On heavily masked training data, the first model had lower MLM loss at 50% masking (2.20 vs 2.67), indicating memorization. At 95% masking it had higher loss (6.23 vs 5.81), which the authors read as weaker grammatical generalization.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.conll-babylm.28

@inproceedings{xiao2023more,
  title = {{Towards more Human-like Language Models based on Contextualizer Pretraining Strategy}},
  author = {Xiao, Chenghao and Hudson, G Thomas and Moubayed, Noura Al},
  booktitle = {Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning},
  year = {2023},
  doi = {10.18653/v1/2023.conll-babylm.28},
  url = {https://doi.org/10.18653/v1/2023.conll-babylm.28}
}

References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tyler A Chang and Benjamin K Bergen. 2022. Word acquisition in neural language models. Transactions of the Association for Computational Linguistics, 10:1– 16. find
  2. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021a. A framework for few-shot language model evaluation. find
  3. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021b. Simcse: Simple contrastive learning of sentence embeddings. In 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, pages 6894–6910. Association for Computational Linguistics (ACL). find our card
  4. Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673. find
  5. Jill Gilkerson, Jeffrey A Richards, Steven F Warren, Judith K Montgomery, Charles R Greenwood, D Kim-brough Oller, John HL Hansen, and Terrance D Paul. 2017. Mapping the early language environment using all-day recordings and automated analysis. American journal of speech-language pathology, 26(2):2… find
  6. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556. find
  7. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  8. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725. find
  9. Daniel Simig, Tianlu Wang, Verna Dankers, Peter Henderson, Khuyagbaatar Batsuren, Dieuwke Hupkes, and Mona Diab. 2022. Text characterization toolkit (tct). In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joi… find
  10. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32. find our card
  11. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Gotlieb Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Adina Williams, Bhargavi Paranjabe, Tal Linzen, and Ryan Cotterell. 2023. Findings of the 2023 BabyLM Challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proce… find
  12. We have also run some experiments on fine-tuning tasks on all model entries, and decided to use "1-40 cnc" for our final submission. Even though "40-1 n" provides the best BLiMP scores, it seems to rely on tasks with unstable behaviours such as irregular forms and quantifiers. find
  13. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020a. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377– 392. find
  14. Alex Warstadt, Yian Zhang, Haau Sing Li, Haokun Liu, and Samuel R Bowman. 2020b. Learning which features matter: Roberta acquires a preference for linguistic generalizations (eventually). In 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, pages 217–235. Association f… find
  15. Chenghao Xiao, Yang Long, and Noura Al Moubayed. 2023. On isotropy, contextualization and learning dynamics of contrastive-based sentence representation learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12266–12283. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiao-2023-more-human-like-language-models/