Elamur Research⌕
☰
home / papers / 2024 / Language Model Behavior: A Comprehensive Survey

Language Model Behavior: A Comprehensive Survey

Paper · 2024 · Tyler A. Chang and Benjamin K. Bergen

Summary

Transformer language models are now deployed widely, often with little or no task-specific tuning, but their outputs remain hard to predict, and the research on their behavior is scattered across hundreds of papers. Chang and Bergen consolidate this evidence by reviewing over 250 behavioral studies of English masked and autoregressive models that were not fine-tuned. These studies treat models as black boxes and probe them with controlled inputs, often borrowing methods from psycholinguistics. The review covers social bias, memorization, reasoning, and knowledge of the world, along with the linguistic areas of syntax, semantics, and pragmatics. Its central conclusion is that models have real but fragile abilities in each of these areas. Performance depends heavily on exact wording and surface cues. Larger models produce much better text, yet they still make factual and commonsense mistakes, reproduce memorized passages, and show biases. The authors suggest that many of these failures can be understood as patterns in text that the model generalizes too broadly or too narrowly. The survey serves as a reference for practitioners and for researchers in neighboring fields. Note that multilingual and non-English models, including Ukrainian, fall outside its scope.

Key points

  • Papers were collected through Semantic Scholar. A seed set of 271 papers led to more than 15K citing works, scraped on February 4, 2023. Screening by title reduced these to about 1,500, and screening by abstract to about 400 highly relevant papers.
  • The scope excludes fine-tuned models, because fine-tuning can latch onto spurious correlations and erode knowledge from pre-training. It also excludes multilingual, vision-language, code, speech, knowledge-augmented, sparse, and compressed models.
  • The models covered range from roughly 100M to 500B parameters, with vocabularies of about 30K–250K tokens. Examples include GPT-3/4, PaLM, OPT, and BERT. The survey also notes when instruction-tuned or RLHF-tuned GPT-3 variants such as text-davinci-003 are involved.
  • The authors pose an open question: do gains from scale come from learning sturdier generalizations, or from memorizing more specific cases? They lean toward the second explanation.
  • Behavioral and mechanistic analyses are related through Marr's levels of analysis. Code and key points are available at github.com/tylerachang/llm-behavior-survey.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California San DiegoSan Diego · United States34 works · 51,167 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Language models & pretraining · 1,343 works

Cite

Cite as Tyler A. Chang and Benjamin K. Bergen. 2024. Language Model Behavior: A Comprehensive Survey. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00492

@article{chang2024language,
  title = {{Language Model Behavior: A Comprehensive Survey}},
  author = {Tyler A. Chang and Benjamin K. Bergen},
  journal = {Computational Linguistics},
  year = {2024},
  doi = {10.1162/coli\_a\_00492},
  url = {https://doi.org/10.1162/coli_a_00492}
}

References

154 references, 53 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdou, Mostafa, Vinit Ravishankar, Artur Kulmizev, and Anders Søgaard. 2022. Word order does matter and shuffled language models know it. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6907–6919. https://doi.org/10.18653/v1 /… find
  2. Aher, Gati, Rosa Arriaga, and Adam Kalai. 2022. Using large language models to simulate multiple humans. ArXiv, arXiv:2208.10264. find
  3. Aina, Laura and Tal Linzen. 2021. The language model understood the prompt was ambiguous: Probing syntactic uncertainty through generation. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 42–57. https://doi.org/10.18653/v1 /2021.blackbox… find
  4. Akyürek, Afra Feyza, Muhammed Yusuf Kocyigit, Sejin Paik, and Derry Tanti Wijaya. 2022. Challenges in measuring bias via open-ended language generation. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 76–76. https://doi.org/10.18653/v1/2022.gebnlp-1… ↗
  5. Aribandi, Vamsi, Yi Tay, and Donald Metzler. 2021. How reliable are model diagnostics? In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1778–1785. https://doi.org/10.18653/v1/2021.findings -acl.155 Armeni, Kristijan, Christopher Honey, and Tal Linzen. 2022. Chara… ↗
  6. Bang, Yejin, Nayeon Lee, Etsuko Ishii, Andrea Madotto, and Pascale Fung. 2021. Assessing political prudence of open-domain chatbots. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 548–555. https://doi.org/10.18653/v1/2021.sigdial-1.57 Bartl,… ↗
  7. Bechtel, William and Oron Shagrir. 2015. The non-redundant contributions of Marr’s three levels of analysis for explaining information-processing mechanisms. find
  8. Topics in Cognitive Science, 7(2):312–322. https://doi.org/10.1111/tops.12141, PubMed: 25900887 Belinkov, Yonatan. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219. https://doi.org/10.1162/coli_a_00422 ↗ our card
  9. Beloucif, Meriem and Chris Biemann. 2021. Probing pre-trained language models for semantic attributes and their values. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2554–2559. https://doi.org/10.18653/v1/2021 .findings-emnlp.218 Bender, Emily M., Timnit Gebru, Ange… ↗
  10. Binz, Marcel and Eric Schulz. 2023. Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences of the United States of America, 120(6):e2218523120. https://doi.org/10.1073/pnas.2218523120, PubMed: 36730192 Blodgett, Su Lin, Solon Barocas, Hal Daumé III, and Han… ↗
  11. Borgeaud, Sebastian, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin… find
  12. Bowman, Samuel. 2022. The dangers of underclaiming: Reasons for caution when reporting how NLP systems fail. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7484–7499. https://doi.org/10.18653/v1/2022.acl-long.516 Brandl, Ste… ↗
  13. Broyde, Joshua and Claire Palmer. 2021. Build a medical sentence matching application using BERT and Amazon SageMaker. find
  14. AWS Machine Learning Blog. Cai, Xingyu, Jiaji Huang, Yu-Lan Bian, and Kenneth Ward Church. 2021. Isotropy in the contextual embedding space: Clusters and manifolds. In International Conference on Learning Representations. find
  15. Cao, Boxi, Hongyu Lin, Xianpei Han, Fangchao Liu, and Le Sun. 2022. Can prompt probe pretrained language models? Understanding the invisible risks from a causal view. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5796–5808.… find
  16. Carlini, Nicholas, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In USENIX Security Symposium, pages 2633–2650. our card
  17. Syntax: A Generative Introduction. Blackwell. Caron, Graham and Shashank Srivastava. 2022. Identifying and manipulating the personality traits of language models. ArXiv, arXiv:2212.10276. https://doi.org/10.18653/v1/2023.findings-emnlp.156 Chang, Tyler, Zhuowen Tu, and Benjamin Bergen. 2022. The ge… ↗
  18. Chaves, Rui P. and Stephanie N. Richter. 2021. Look at that! BERT can be easily distracted from paying attention to morphosyntax. In Proceedings of the Society for Computation in Linguistics 2021, pages 28–38. find
  19. Chen, Kaiping, Anqi Shao, Jirayu Burapacheep, and Yixuan Li. 2022. A critical appraisal of equity in conversational AI: Evidence from auditing GPT-3’s dialogues with different publics on climate change and Black Lives Matter. ArXiv, arXiv:2209.13627. — [arXiv:2209.13627](https://arxiv.org/abs/2209.13627) find
  20. Chen, Mark, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alet… our card
  21. Chiang, Cheng Han, Sung-Feng Huang, and Hung-yi Lee. 2020. Pretrained language model embryology: The birth of ALBERT. find
  22. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6813–6828. https://doi.org/10.18653/v1/2020.emnlp-main.553 Chiang, Ting Rui and Yun-Nung Chen. 2021. Relating neural text degeneration to exposure bias. In Proceedings of the Fourth BlackboxNLP… ↗ our card
  23. Chuang, Chengyu and Yi Yang. 2022. Buy Tesla, sell Ford: Assessing implicit stock market preference in pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 100–105. https://doi.org /10.18653/v1/2022.ac… ↗
  24. Cong, Yan. 2022. Psycholinguistic diagnosis of language models’ commonsense reasoning. In Proceedings of the First Workshop on Commonsense Representation and Reasoning (CSRR 2022), pages 17–22. https://doi.org/10.18653/v1/2022.csrr-1.3 Czarnowska, Paula, Yogarshi Vyas, and Kashif Shah. 2021. Quanti… ↗
  25. Czinczoll, Tamara, Helen Yannakoudakis, Pushkar Mishra, and Ekaterina Shutova. 2022. Scientific and creative analogies in pretrained language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2094–2100. https://doi.org/10.18653/v1/2022 .findings-emnlp.153 Dabrow… ↗ our card
  26. Dar, Guy, Mor Geva, Ankit Gupta, and Jonathan Berant. 2022. Analyzing Transformers in embedding space. ArXiv, arXiv:2209.02535. https://doi.org/10.18653/v1/2023.acl-long.893 Dasgupta, Ishita, Andrew Lampinen, Stephanie Chan, Antonia Creswell, Dharshan Kumaran, James McClelland, and Felix Hill. 2022… ↗ our card
  27. Davis, Forrest and Marten van Schijndel. 2020. Discourse structure interacts with reference but not syntax in neural language models. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 396–407. https://doi.org/10.18653/v1/2020.conll-1.32 Davison, Joe, Joshua Fel… ↗
  28. De Bruyn, Maxime, Ehsan Lotfi, Jeska Buhmann, and Walter Daelemans. 2022. Is it smaller than a tennis ball? Language models play the game of twenty questions. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 80–90. https://doi.org/10.18653… find
  29. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… our card
  30. Dhamala, Jwala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. BOLD: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency, pages 862–872. ht… find
  31. Du, Yifan, Zikang Liu, Junyi Li, and Wayne Xin Zhao. 2022b. A survey of vision-language pre-trained models. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 5436–5443. https://doi.org/10.24963/ijcai.2022/762 Dufter, Philipp, Martin Schmitt, and Hinrich Schütze… ↗
  32. Dugan, Liam, Daphne Ippolito, Arun Kirubarajan, Sherry Shi, and Chris Callison-Burch. 2023. Real or fake text?: Investigating human ability to detect boundaries between human-written and machine-generated text. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 12763–12771. http… find
  33. Fedus, William, Barret Zoph, and Noam Shazeer. 2022. Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23:1–39. find
  34. Felkner, Virginia K., Ho-Chun Herbert Chang, Eugene Jang, and Jonathan May. 2022. Towards WinoQueer: Developing a benchmark for anti-queer bias in large language models. In Queer in AI Workshop. https://doi.org/10.18653/v1/2023.acl-long.507 Finlayson, Matthew, Aaron Mueller, Sebastian Gehrmann, Stu… ↗
  35. Freitag, Markus and Yaser Al-Onaizan. 2017. Beam search strategies for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 56–60. https://doi.org/10 .18653/v1/W17-3207 Fricke, Suzanne. 2018. Semantic Scholar. Journal of the Medical Library Associatio… our card
  36. Ganguli, Deep, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Scott Johnston, Andy Jones, Nicholas Joseph, Jackson Kernian, Shauna Kravec, Ben Mann, Neel Nan… find
  37. Gauthier, Jon, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. SyntaxGym: An online platform for targeted evaluation of language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70–76. https://doi.org/10.1865… our card
  38. Geeraerts, Dirk. 2017. Lexical semantics. Oxford Research Encyclopedia of Linguistics. https://doi.org/10.1093/acrefore/9780199384655.013.29 Gehman, Samuel, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language mod… ↗ our card
  39. Grice, H. P. 1975. Logic and conversation. Syntax and Semantics: Vol. 3: Speech Acts, pages 41–58. https://doi.org/10.1163/9789004368811 003 Griciūtė, Bernadeta, Marc Tanti, and Lucia Donatelli. 2022. On the cusp of comprehensibility: Can language models distinguish between metaphors and nonsense… ↗
  40. Hagendorff, Thilo, Sarah Fabi, and Michal Kosinski. 2022. Machine intuition: Uncovering human-like intuitive decision-making in GPT-3.5. ArXiv, arXiv:2212.05206. https://doi.org/10.1038/s43588-023-00527-x Hahn, Michael. 2020. Theoretical limitations of self-attention in neural sequence models. Tran… ↗
  41. Han, Simeng, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, David Peng, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Shafiq Joty, Alexander R. Fabbri, Wo… find
  42. Hendrycks, Dan, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track. our card
  43. Hernandez, Danny, Tom B. Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Benjamin Mann, Christopher Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandlish. 2022. Scaling law… our card
  44. Hershcovich, Daniel, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges and strategies in cross-cultural… our card
  45. Holtzman, Ari, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations. find
  46. Hossain, Md Mosharaf, Dhivya Chinnappa, and Eduardo Blanco. 2022. An analysis of negation in natural language understanding corpora. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 716–723. https://doi.org /10.18653/v1/2022.a… ↗
  47. Hu, Jennifer, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2022. A fine-grained comparison of pragmatic language understanding in humans and language models. ArXiv, arXiv:2212.06801. https://doi.org/10.18653/v1/2023.acl-long.230 Hu, Jennifer, Jon Gauthier, Peng Qian, Ethan… ↗ our card
  48. Huynh, Hien, Tomas O. Lentz, and Emiel van Miltenburg. 2022. Implicit causality in GPT-2: A case study. ArXiv, arXiv:2212.04348. find
  49. Ippolito, Daphne, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2020. Automatic detection of generated text is easiest when humans are fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1808–1822. https://doi.org /10.18653/v1/2020.acl… ↗ our card
  50. Jain, Sarthak and Byron C. Wallace. 2019. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543–3556. our card
  51. Jakesch, Maurice, Jeffrey T. Hancock, and Mor Naaman. 2023. Human heuristics for AI-generated language are flawed. Proceedings of the National Academy of Sciences, 120(11):e2208839120. https://doi.org/10.1073/pnas.2208839120 PubMed: 36881628 Jang, Joel, Seonghyeon Ye, and Minjoon Seo. 2022. Can lar… ↗
  52. Jawahar, Ganesh, Muhammad Abdul-Mageed, and Laks Lakshmanan, V. S. 2020. Automatic detection of machine generated text: A critical survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2296–2309. https://doi.org/10.18653/v1/2020.coling-main.208 Jiang, Cheng… ↗ our card
  53. Jiang, Guangyuan, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. 2022. MPI: Evaluating and inducing personality in pre-trained language models. ArXiv, arXiv:2206.07550. find
  54. Jiang, Tianyu and Ellen Riloff. 2021. Learning prototypical functions for physical artifacts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6941–6951… find
  55. Jin, Zijia, Xingyu Zhang, Mo Yu, and Lifu Huang. 2022b. Probing script knowledge from pre-trained models. In Proceedings of the Workshop on Unimodal and Multimodal Induction of Linguistic Structures (UM-IoS), pages 87–93. https://doi.org/10.18653/v1/2022.umios-1.10 Johnson, Rebecca L, Giada Pistill… ↗
  56. Johnson, Steven. 2022. A.I. is mastering language. Should we trust what it says? The New York Times. find
  57. Jones, Cameron and Benjamin Bergen. 2021. The role of physical inference in pronoun resolution. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 43, pages 2876–2882. find
  58. Jones, Cameron R., Tyler A. Chang, Seana Coulson, James Michaelov, Sean Trott, and Benjamin Bergen. 2022. Distributional semantics still can’t account for affordances. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 44, pages 482–489. find
  59. Joshi, Pratik, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293. https://doi.org /10.18653/v… ↗ our card
  60. Kadavath, Saurav, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hern… find
  61. Kalo, Jan Christoph and Leandra Fichtel. 2022. KAMEL: Knowledge analysis with multitoken entities in language models. In 4th Conference on Automated Knowledge Base Construction. find
  62. Kandpal, Nikhil, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2022. Large language models struggle to learn long-tail knowledge. ArXiv, arXiv:2211.08411. our card
  63. Kandpal, Nikhil, Eric Wallace, and Colin Raffel. 2022. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pages 10697–10707. our card
  64. Kaplan, Jared, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. 2020. Scaling laws for neural language models. ArXiv, arXiv:2001.08361. our card
  65. Karpas, Ehud, Omri Abend, Yonatan Belinkov, Barak Lenz, Opher Lieber, Nir Ratner, Yoav Shoham, Hofit Bata, Yoav Levine, Kevin Leyton-Brown, Dor Muhlgay, Noam Rozen, Erez Schwartz, Gal Shachaf, Shai Shalev-Shwartz, Amnon Shashua, and Moshe Tenenholtz. 2022. MRKL Systems: A modular, neuro-symbolic arc… our card
  66. Kavumba, Pride, Ryo Takahashi, and Yusuke Oda. 2022. Are prompt-based models clueless? In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2333–2352. https://doi.org/10.18653/v1/2022.acl-long.166 Kementchedjhieva, Yova, Mark Ande… ↗
  67. Kharitonov, Eugene, Marco Baroni, and Dieuwke Hupkes. 2021. How BPE affects memorization in Transformers. ArXiv, arXiv:2110.02782. — [arXiv:2110.02782](https://arxiv.org/abs/2110.02782) find
  68. Kim, Sanghee J., Lang Yu, and Allyson Ettinger. 2022. “no, they did not”: Dialogue response dynamics in pre-trained language models. In Proceedings of the 29th International Conference on Computational Linguistics, pages 863–874. find
  69. Kirchenbauer, John, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. A watermark for large language models. ArXiv, arXiv:2301.10226. our card
  70. Kirk, Hannah Rose, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano. 2021. Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models. In Advances in Neural Information Process… find
  71. Ko, Wei Jen and Junyi Jessy Li. 2020. Assessing discourse relations in language generation from GPT-2. In Proceedings of the 13th International Conference on Natural Language Generation, pages 52–59. https://doi.org/10.18653/v1/2020.inlg-1.8 Kojima, Takeshi, Shixiang Shane Gu, Machel Reid, Yutaka M… ↗ our card
  72. Kovaleva, Olga, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4365–43… our card
  73. The Keyword: Google Blog. Kudo, Taku. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75. https://doi.org/10.18653/… our card
  74. Kwon, Sunjae, Cheongwoong Kang, Jiyeon Han, and Jaesik Choi. 2019. Why do masked neural language models still need common sense knowledge? ArXiv, arXiv:1911.03024. — [arXiv:1911.03024](https://arxiv.org/abs/1911.03024) find
  75. Lakhotia, Kushal, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics,… our card
  76. Lal, Yash Kumar, Niket Tandon, Tanvi Aggarwal, Horace Liu, Nathanael Chambers, Raymond Mooney, and Niranjan Balasubramanian. 2022. Using commonsense knowledge to answer why-questions. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1204–1219. https://… find
  77. Lampinen, Andrew, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. 2022. Can language models learn from explanations in context? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 537–563. https:/… our card
  78. Lee, Angie. 2023. What are large language models used for? NVIDIA Blog. find
  79. Lee, Jooyoung, Thai Le, Jinghui Chen, and Dongwon Lee. 2023. Do language models plagiarize? In The ACM Web Conference, pages 3637–3647. https://doi.org/10.1145/3543507.3583199 Lee, Katherine, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 20… ↗
  80. Lees, Alyssa, Vinh Q. Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman. 2022. A new generation of perspective API: Efficient multilingual character-level Transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3197–320… find
  81. Lehman, Eric, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron Wallace. 2021. Does BERT pretrained on clinical notes reveal sensitive data? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 94… find
  82. Li, Junyi, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen. 2021. Pretrained language models for text generation: A survey. In International Joint Conference on Artificial Intelligence, pages 4492–4499. https://doi.org/10.24963/ijcai.2021/612 Li, Xiang Lorraine, Adhiguna Kuncoro, Jordan Hoffmann, Cypr… ↗
  83. Lieber, Opher, Or Sharir, Barak Lenz, and Yoav Shoham. 2021. Jurassic-1: Technical details and evaluation. White Paper. AI21 Labs. find
  84. Lin, Bill Yuchen, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020. Birds have four legs?! NumerSense: Probing numerical commonsense knowledge of pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6862–6868. https://doi.… ↗ our card
  85. Liu, Ruibo, Chenyan Jia, Jason Wei, Guangxuan Xu, and Soroush Vosoughi. 2022c. Quantifying and alleviating political bias in language models. Artificial Intelligence, 304:103654. https://doi.org/10.1016/j.artint.2021.103654 Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, O… ↗
  86. Liu, Zeyu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A. Smith. 2021. Probing across time: What does RoBERTa know and when? In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 820–842. https://doi.org/10.18653/v1/2021.findings-emnlp.71 Magee, Liam, Lida Gha… ↗ our card
  87. Mahowald, Kyle. 2023. A discerning several thousand judgments: GPT-3 rates the article + adjective + numeral + noun construction. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 265–273. https://doi.org /10.18653/v1/2023.eacl-main… ↗
  88. Malkin, Nikolay, Sameera Lanka, Pranav Goel, and Nebojsa Jojic. 2021. Studying word order through iterative shuffling. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10351–10366. https://doi.org/10.18653/v1/2021.emnlp-main.809 Mallen, Alex, Akari As… ↗
  89. Mattern, Justus, Zhijing Jin, Mrinmaya Sachan, Rada Mihalcea, and B. Schölkopf. 2022. Understanding stereotypes in language models: Towards robust measurement and zero-shot debiasing. ArXiv, arXiv:2212.10678. — [arXiv:2212.10678](https://arxiv.org/abs/2212.10678) find
  90. McCoy, R. Thomas, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. 2021. How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN. ArXiv, arXiv:2111.09509. our card
  91. McCoy, Thomas, Robert Frank, and Tal Linzen. 2018. Revisiting the poverty of the stimulus: Hierarchical generalization without a hierarchical bias in recurrent neural networks. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 40, pages 2096–2101. find
  92. McCoy, Tom, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448. https://doi.org/10.18653/v1/P19 -1334 Meade, Nicho… ↗ our card
  93. Miaschi, Alessio, Dominique Brunato, Felice Dell’Orletta, and Giulia Venturi. 2020. Linguistic profiling of a neural language model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 745–756. https://doi.org/10.18653/v1/2020.coling-main.65 Michaelov, James and… ↗
  94. Misra, Kanishka, Allyson Ettinger, and Julia Rayz. 2020. Exploring BERT’s sensitivity to lexical cues using tests from semantic priming. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4625–4635. https://doi.org/10.18653/v1/2020.findings-emnlp.415 Misra, Kanishka, Al… ↗
  95. Misra, Kanishka, Julia Taylor Rayz, and Allyson Ettinger. 2023. COMPS: Conceptual minimal pair sentences for testing property knowledge and inheritance in pre-trained language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, page… find
  96. Mosbach, Marius, Anna Khokhlova, Michael A. Hedderich, and Dietrich Klakow. 2020. On the interplay between fine-tuning and sentence-level probing for linguistic knowledge in pre-trained Transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2502–2516. https://d… find
  97. The Keyword: Google Blog. Newman, Benjamin, Kai-Siang Ang, Julia Gong, and John Hewitt. 2021. Refining targeted syntactic evaluation of language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  98. Nozza, Debora, Federico Bianchi, and Dirk Hovy. 2021. HONEST: Measuring hurtful sentence completion in language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2398–2406. https://doi.org/… find
  99. OpenAI. 2022. ChatGPT: Optimizing language models for dialogue. find
  100. OpenAI Blog. OpenAI. 2023a. GPT-4 technical report. our card
  101. OpenAI. OpenAI. 2023b. Model index for researchers. find
  102. OpenAI. Ousidhoum, Nedjma, Xinran Zhao, Tianqing Fang, Yangqiu Song, and Dit-Yan Yeung. 2021. Probing toxic content in large pre-trained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural… find
  103. Patel, Roma and Ellie Pavlick. 2021. “was it “stated” or was it “claimed”?: How linguistic bias affects generative language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10080–10095. https://doi.org/10.18653/v1/2021.emnlp-main.790 Pedinotti… ↗ our card
  104. Petty, Jackson and Robert Frank. 2021. Transformers generalize linearly. ArXiv, arXiv:2109.12036. — [arXiv:2109.12036](https://arxiv.org/abs/2109.12036) find
  105. Piantadosi, Steven and Evelina Fedorenko. 2017. Infinitely productive language can arise from chance under communicative pressure. Journal of Language Evolution, 2(2):141–147. https://doi.org/10.1093/jole/lzw013 Podkorytov, Maksim, Daniel Bis, and Xiuwen Liu. 2021. How can the [MASK] know? The sour… ↗
  106. Porada, Ian, Alessandro Sordoni, and Jackie Cheung. 2022. Does pre-training induce systematic inference? How masked language models acquire commonsense knowledge. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech… find
  107. Press, Ofir, Noah Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations. find
  108. Press, Ofir, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. ArXiv, arXiv:2210.03350. https://doi.org/10.18653/v1/2023.findings-emnlp.378 Qin, Lianhui, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Cho… ↗
  109. Radford, Alec, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  110. OpenAI. Radford, Alec, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI. Rae, Jack W., Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susa… find
  111. Raffel, Colin, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text Transformer. Journal of Machine Learning Research, 21(140):5485–5551. our card
  112. Raj, Harsha, Domenic Rosati, and Subhabrata Majumdar. 2022. Measuring reliability of large language models through semantic consistency. In NeurIPS ML Safety Workshop. find
  113. Ravichander, Abhilasha, Eduard Hovy, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Cheung. 2020. On the systematicity of probing contextualized word representations: The case of hypernymy in BERT. In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pages 88–102. find
  114. Razeghi, Yasaman, Robert Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 840–854. https://doi.org/10.18653/v1/2022.findings-emnlp.59 Reif, Emily, D… ↗ our card
  115. Rogers, Anna, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866. https://doi.org/10.1162/tacl_a_00349 ↗ our card
  116. Romero, Julien and Simon Razniewski. 2022. Do children texts hold the key to commonsense knowledge? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10954–10959. https://doi.org/10.18653/v1/2022.emnlp-main.752 Ruis, Laura, Akbir Khan, Stella Rose Bide… ↗
  117. Ryu, Soo Hyun and Richard Lewis. 2021. Accounting for agreement phenomena in sentence comprehension with transformer language models: Effects of similarity-based interference on surprisal and attention. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 61–71.… find
  118. Sancheti, Abhilasha and Rachel Rudinger. 2022. What do large language models learn about scripts? In Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 1–11. https://doi.org/10.18653/v1/2022.starsem-1.1 Sanh, Victor, Lysandre Debut, Julien Chaumond, and Thomas Wo… ↗
  119. Sap, Maarten, Ronan Le Bras, Daniel Fried, and Yejin Choi. 2022. Neural theory-of-mind? On the limits of social intelligence in large LMs. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3762–3780. https://doi.org/10.18653/v1/2022.emnlp-main.248 Sapa… ↗
  120. Schuster, Sebastian and Tal Linzen. 2022. When a sentence does not introduce a discourse entity, transformer-based models still sometimes refer to it. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pa… find
  121. Shaikh, Omar, Hongxin Zhang, William B. Held, Michael Bernstein, and Diyi Yang. 2022. On second thought, let’s not think step by step! Bias and toxicity in zero-shot reasoning. ArXiv, arXiv:2212.08061. https://doi.org/10.18653/v1/2023 .acl-long.244 Shardlow, Matthew and Piotr Przybyla. 2022. Deanthr… ↗ our card
  122. Shaw, Peter, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464–468. ht… our card
  123. Shi, Haohan and Phillip Wolff. 2021. What Transformers might know about the physical world: T5 and the origins of knowledge. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 43, pages 2218–2224. find
  124. Shwartz, Vered and Yejin Choi. 2020. Do neural language models overcome reporting bias? In Proceedings of the 28th International Conference on Computational Linguistics, pages 6863–6870. https://doi.org/10.18653/v1/2020.coling-main.605 Shwartz, Vered, Rachel Rudinger, and Oyvind Tafjord. 2020. “you… ↗
  125. Spitale, Giovanni, Nikola Biller-Andorno, and Federico Germani. 2023. AI model GPT-3 (dis)informs us better than humans. ArXiv, arXiv:2301.11924. https://doi.org/10.1126/sciadv.adh1850, PubMed: 37379395 Srivastava, Aarohi, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch,… ↗ our card
  126. Stammbach, Dominik, Maria Antoniak, and Elliott Ash. 2022. Heroes, villains, and victims, and GPT-3: Automated extraction of character roles without training data. In Proceedings of the 4th Workshop of Narrative Understanding (WNU2022), pages 47–56. https://doi.org/10.18653/v1/2022.wnu-1.6 ↗
  127. Stevenson, Claire, Iris Smal, Matthijs Baas, Raoul Grasman, and Han van der Maas. 2022. Putting GPT-3’s creativity to the (alternative uses) test. In International Conference on Computational Creativity, pages 164–168. Stolfo, Alessandro, Zhijing Jin, Kumar Shridhar, Bernhard Schölkopf, and Mrinmay… our card
  128. Su, Jianlin, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2021. RoFormer: Enhanced Transformer with rotary position embedding. ArXiv, arXiv:2104.09864. our card
  129. Summers-Stay, Douglas, Claire Bonial, and Clare Voss. 2021. What can a generative language model answer about a passage? In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 73–81. https://doi.org/10.18653/v1/2021.mrqa-1.7 Suzgun, Mirac, Nathan Scales, Nathanael Schar… ↗ our card
  130. Tal, Yarden, Inbal Magar, and Roy Schwartz. 2022. Fewer errors, but more stereotypes? The effect of model size on gender bias. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 112–120. https://doi.org/10.18653/v1/2022.gebnlp-1.13 Tamborrino, Alexandre… ↗
  131. Tay, Yi, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, W. Fedus, Jinfeng Rao, Sharan Narang, Vinh Tran, Dani Yogatama, and Donald Metzler. 2022a. Scaling laws vs model architectures: How does inductive bias influence scaling? ArXiv, arXiv:2207.10551. https://doi.org/10.18653/v1 /2023.findings-emn… our card
  132. Tejada, Giorgia Nidia Carranza, Johannes Scholtes, and Gerasimos Spanakis. 2021. A study of BERT’s processing of negations to determine sentiment. Benelux Conference on Artificial Intelligence and the Belgian Dutch Conference on Machine Learning, pages 47–59. find
  133. Tenney, Ian, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601. https://doi.org/10.18653/v1/P19-1452 ↗ our card
  134. Thewsey, Alex. 2021. Bring structure to diverse documents with Amazon Textract and transformer-based models on Amazon SageMaker. find
  135. AWS Machine Learning Blog. Tirumala, Kushal, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022. Memorization without overfitting: Analyzing the training dynamics of large language models. In Advances in Neural Information Processing Systems, volume 35, pages 38274–38290. our card
  136. Touileb, Samia. 2022. Exploring the effects of negation and grammatical tense on bias probes. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short… find
  137. Traylor, Aaron, Roman Feiman, and Ellie Pavlick. 2021. AND does not mean OR: Using formal languages to study language models’ representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pr… find
  138. Blog Post. Trott, Sean, Cameron J. Jones, Tyler A. Chang, James Michaelov, and Benjamin Bergen. 2023. Do large language models know what humans know? Cognitive Science, 47(7):e13309. https://doi.org/10.1111/cogs.13309, PubMed: 37401923 Truong, Thinh Hung, Yulia Otmakhova, Timothy Baldwin, Trevor Co… ↗ our card
  139. Tuckute, Greta, Aalok Sathe, Mingye Wang, Harley Yoder, Cory Shain, and Evelina Fedorenko. 2022. SentSpace: Large-scale benchmarking and evaluation of text using cognitively motivated lexical, syntactic, and semantic features. In Proceedings of the 2022 Conference of the North American Chapter of th… find
  140. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998–6008. our card
  141. Vig, Jesse and Yonatan Belinkov. 2019. Analyzing the structure of attention in a transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63–76. https://doi.org/10.18653/v1/W19 -4808 Vig, Jesse, Sebastian Gehrmann, Yon… ↗ our card
  142. Wahle, Jan Philip, Terry Ruas, Frederic Kirstein, and Bela Gipp. 2022. How large language models are transforming machine-paraphrase plagiarism. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 952–963. https://doi.org/10.18653/v1/2022.emnlp-main.62 W… ↗
  143. Wang, Boshi, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2022a. Towards understanding chain-of-thought prompting: An empirical study of what matters. ArXiv, arXiv:2212.10001. https://doi.org/10.18653/v1/2023.acl-long.153 Wang, Benyou, Lifeng Shang, Christina Lioma,… ↗ our card
  144. Wang, Cunxiang, Boyuan Zheng, Yuchen Niu, and Yue Zhang. 2021b. Exploring generalization ability of pretrained language models on arithmetic and logical reasoning. In Natural Language Processing and Chinese Computing, pages 758–769. https://doi.org/10.1007/978-3-030 -88480-2 61 Wang, Tianlu, Rohit S… ↗
  145. Wei, Jason, Dan Garrette, Tal Linzen, and Ellie Pavlick. 2021. Frequency effects on syntactic rule learning in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 932–948. https://doi.org/10.18653/v1/2021.emnlp-main.72 Wei, Jason, Najoung K… ↗ our card
  146. Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022c. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. our card
  147. Weidinger, Laura, John F. J. Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zachary Kenton, Sande Minnich Brown, William T. Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendric… our card
  148. Weidinger, Laura, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Se… find
  149. Linguistic Inquiry, pages 1–88. https://doi.org/10.1162/ling a 00491 Williams, Adina, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Co… ↗
  150. Zhang, Lining, Mengchen Wang, Liben Chen, and Wenxin Zhang. 2022a. Probing GPT-3’s linguistic knowledge on semantic tasks. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 297–304. https://doi.org/10.18653/v1/2022.blackboxnlp-1.24 Zhang,… ↗
  151. Zhang, Sheng, Xin Zhang, Weiming Zhang, and Anders Søgaard. 2021a. Sociolectal analysis of pretrained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4581–4588. https://doi.org/10.18653/v1/2021.emnlp-main.375 Zhang, Tianyi, Varsha Ki… ↗
  152. Zhang, Yian, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. 2021b. When do you need billions of words of pretraining data? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume… find
  153. Zhang, Zhengyan, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441–1451. https://doi.org/10.18653/v1/P19 -1139 Zhao,… ↗
  154. Zwaan, Rolf A. 2016. Situation models, mental simulations, and abstract concepts in discourse comprehension. Psychonomic Bulletin & Review, 23:1028–1034. https://doi.org/10.3758/s13423-015-0864-x, PubMed: 26088667 — [doi:10.3758/s13423-015-0864-x](https://doi.org/10.3758/s13423-015-0864-x) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chang-2024-language-model-behavior-comprehensive-survey/