Elamur Research⌕
☰
home / papers / 2024 / Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluation

Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluation

Paper · 2024 · Cyril Chhun, Fabian M. Suchanek and Chloé Clavel

Summary

Rating machine-written stories by hand is slow and expensive. Standard automatic metrics such as BLEU, ROUGE, BERTScore and BARTScore also agree poorly with human judges. The authors test whether LLMs can replace human raters for Automatic Story Evaluation (ASE). They use the HANNA benchmark and its six criteria: Relevance, Coherence, Empathy, Surprise, Engagement and Complexity. LLMs score 1,056 stories on a 1–5 scale under four Eval-Prompt designs: a bare rating, a rating with a justification, a justification plus the original annotation guidelines, and a justification plus a reference human story. The study compares LLM scores with human annotations and existing metrics, checks how consistent the models are, and runs a user study on the quality of their explanations. It then uses LLM raters to judge stories generated by Llama models. Overall, LLM ratings agree only moderately or weakly with humans, but at the system level the agreement is very high. This makes LLMs the best current proxy for ranking story generators, although their justifications often lack well-supported reasoning.

Key points

  • The released resource contains about 150k LLM ratings and explanations (1,056 stories × 6 criteria × 4 Eval-Prompts × 3 tries × 2 models). It also includes 1,500 human judgments of LLM explanations and 384 new Llama-generated stories that extend HANNA. Everything is available on GitHub (dig-team/hanna-benchmark-asg).
  • LLMs give highly repeatable scores across tries, with very high intra-class correlation.
  • Adding detailed guidelines (Eval-Prompt 3) did not improve agreement with human ratings. Supplying a human reference story (Eval-Prompt 4) gave mixed results.
  • Stories written by LLMs receive ratings at least equal to human stories, and larger models score best. A Min-K% Prob analysis on WritingPrompts and BooksMIA suggests that larger models produce text closer to their pretraining data, such as existing books.
  • Source-available Llama models performed at least as well as the proprietary ChatGPT. The authors recommend favouring open models for transparency and reproducibility.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Institut national de recherche en sciences et technologies du numériqueLe Chesnay-Rocquencourt · France84 works · 25,414 citations🔬ALMANACH: Modélisation et analyse linguistique automatique et humanités computationnellesParis · France19 works · 6,924 citations🔬Laboratoire Traitement et Communication de l’InformationPalaiseau · France8 works · 378 citations🎓Télécom ParisPalaiseau · France11 works · 826 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Cyril Chhun, Fabian M. Suchanek and Chloé Clavel. 2024. Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluation. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00689

@article{chhun2024do,
  title = {{Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluation}},
  author = {Cyril Chhun, Fabian M. Suchanek and Chloé Clavel},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.1162/tacl\_a\_00689},
  url = {https://doi.org/10.1162/tacl_a_00689}
}

References

97 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amal Alabdulkarim, Siyan Li, and Xiangyu Peng. 2021. Automatic story generation: Challenges and attempts. In Proceedings of the Third Workshop on Narrative Understanding, pages 72–83, Virtual. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2021.nuse-1.8
  2. Valentin Amrhein, Sander Greenland, and Blake McShane. 2019. Scientists rise up against statistical significance. Nature, 567(7748):305–307. https://doi.org/10.1038/d41586-019 -00857-9, PubMed: 30894741 — [doi:10.1038/d41586-019](https://doi.org/10.1038/d41586-019) ↗
  3. Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christopher Re. 2023. Ask me anything: A simple strategy for prompting language models. In The Eleventh International Conference on Learning Representations. find
  4. Byung-Chull Bae, Suji Jang, Youngjune Kim, and Seyoung Park. 2021. A preliminary survey on story interestingness: Focusing on cognitive and emotional interest. In International Conference on Interactive Digital Storytelling, pages 447–453. Springer. https://doi.org/10.1007/978-3-030-92300-6 45 — [doi:10.1007/978-3-030-92300-6](https://doi.org/10.1007/978-3-030-92300-6) ↗
  5. Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1):289–300. https://doi.org/10.1111/j.2517-6161.1995 .tb02031.x Manik Bhandari, Pranav Naraya… ↗
  6. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  7. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv prep…
  8. Ricardo Campos, Vı́tor Mangaravite, Arian Pasquali, Alı́pio Jorge, Célia Nunes, and Adam Jatowt. 2020. Yake! Keyword extraction from single documents using multiple local features. Information Sciences, 509:257–289. https://doi.org/10.1016/j.ins.2019.09.013 — [doi:10.1016/j.ins.2019.09.013](https://doi.org/10.1016/j.ins.2019.09.013) ↗
  9. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. ArXiv preprint, abs/2006.14799v2. find
  10. Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. 2023. Art or artifice? Large language models and the false promise of creativity. arXiv preprint arXiv:2309.14556v1. — [arXiv:2309.14556](https://arxiv.org/abs/2309.14556) find
  11. Cyril Chhun, Pierre Colombo, Fabian M. Suchanek, and Chloé Clavel. 2022. Of human criteria and automatic metrics: A benchmark of the evaluation of story generation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5794–5836, Gyeongju, Republic of Korea. Intern… find
  12. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — PaLM: Scaling Language Modeling with Pathways
  13. Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. 2021. All that’s ‘human’ is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International… find
  14. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. https://doi.org/10.1177/001316446002000104 — [doi:10.1177/001316446002000104](https://doi.org/10.1177/001316446002000104) ↗
  15. Pierre Colombo, Maxime Peyrard, Nathan Noiry, Robert West, and Pablo Piantanida. 2023. The glass ceiling of automatic evaluation in natural language generation. In Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings), pages 178–183, Nusa Dua, Bali. Association for C… find
  16. Pierre Colombo, Guillaume Staerman, Chloé Clavel, and Pablo Piantanida. 2021. Automatic text evaluation through the lens of Wasserstein barycenters. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10450–10466, Online and Punta Cana, Dominican Republi… find
  17. Daniel Deutsch, Rotem Dror, and Dan Roth. 2022. On the limitations of reference-free evaluations of generated text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10960–10977, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics… find
  18. Robert Dickman. 2003. The four elements of every successful story. Reflections - Society for Organizational Learning, 4(3):51–58. https://doi.org/10.1162/15241730360580212 — [doi:10.1162/15241730360580212](https://doi.org/10.1162/15241730360580212) ↗
  19. Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023. Is GPT-3 a good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11173–11195, Toronto, Canada. Association for…
  20. Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A. Smith, and Yejin Choi. 2022. Is GPT-3 text indistinguishable from human text? Scarecrow: A framework for scrutinizing machine text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper…
  21. Igor Douven. 2018. A Bayesian perspective on Likert scales and central tendency. Psychonomic Bulletin & Review, 25:1203–1211. https://doi.org/10.3758/s13423 -017-1344-2, PubMed: 28752379 — [doi:10.3758/s13423](https://doi.org/10.3758/s13423) ↗
  22. Edilivre. 2023. Concours de nouvelles 2023. Accessed: 2023-10-12. find
  23. Majigsuren Enkhsaikhan, Wei Liu, Eun-Jung Holden, and Paul Duuring. 2021. Auto-labelling entities in low-resource text: A geological case study. Knowledge and Information Systems, 63:695–715. https://doi.org/10.1007/s10115-020-01532-6 — [doi:10.1007/s10115-020-01532-6](https://doi.org/10.1007/s10115-020-01532-6) ↗
  24. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia. Association for Computational Linguistics. https://doi.org /1… ↗
  25. Aris Fergadis and Benedikt Scheffler. 2022. Chance-corrected agreement coefficients. find
  26. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378. https://doi.org/10 .1037/h0031619 find
  27. Yang Gao, Wei Zhao, and Steffen Eger. 2020. SUPERT: Towards new frontiers in unsupervised evaluation metrics for multi-document summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1347–1354, Online. Association for Computational Linguistic… find
  28. Daniel R. George, Heather L. Stuckey, and Megan M. Whitehead. 2014. How a creative storytelling intervention can improve medical student attitude towards persons with dementia: A mixed methods study. Dementia, 13(3):318–329, https://doi.org/10 .1177/1471301212468732, PubMed: 24770946 find
  29. Seraphina Goldfarb-Tarrant, Tuhin Chakrabarty, Ralph Weischedel, and Nanyun Peng. 2020. Content planning for neural story generation with Aristotelian rescoring. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4319–4338, Online. Association fo… find
  30. Yvette Graham and Timothy Baldwin. 2014. Testing for significance of increased correlation with human judgment. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 172–176, Doha, Qatar. Association for Computational Linguistics. https://doi .org/1…
  31. Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, and Minlie Huang. 2020. A knowledge-enhanced pretraining model for commonsense story generation. Transactions of the Association for Computational Linguistics, 8:93–108. https://doi.org/10.1162/tacl a 00302 ↗
  32. Jian Guan, Xiaoxi Mao, Changjie Fan, Zitao Liu, Wenbiao Ding, and Minlie Huang. 2021. Long text generation by modeling sentence-level and discourse-level coherence. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference… find
  33. Kilem Li Gwet. 2008. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29–48. https://doi.org/10.1348/000711006X126600, PubMed: 18482474 ↗
  34. Kevin A. Hallgren. 2012. Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8(1):23. https://doi.org/10.20982/tqmp.08.1.p023, PubMed: 22833776 — [doi:10.20982/tqmp.08.1.p023](https://doi.org/10.20982/tqmp.08.1.p023) ↗
  35. Andrew F. Hayes and Klaus Krippendorff. 2007. Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1):77–89. https://doi.org/10.1080/19312450 709336664 — [doi:10.1080/19312450](https://doi.org/10.1080/19312450) ↗
  36. Mohieddin Jafari and Naser Ansari-Pour. 2019. Why, when and how to adjust your P values? Cell Journal (Yakhteh), 20(4):604–607. https://doi.org/10.22074/cellj .2019.5992 — [doi:10.22074/cellj](https://doi.org/10.22074/cellj) ↗
  37. João Ricardo de Oliveira Júnior, Ricardo Limongi, Weng Marc Lim, Jacqueline K. Eastman, and Satish Kumar. 2023. A story to sell: The influence of storytelling on consumers’ purchasing behavior. Psychology & Marketing, 40(2):239–261. https://doi.org/10.1002/mar.21758 — [doi:10.1002/mar.21758](https://doi.org/10.1002/mar.21758) ↗
  38. Daniel Kahneman. 2011. Thinking, Fast and Slow. Farrar, Straus and Giroux. find
  39. Maurice G. Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81–93. https://doi.org/10.2307/2332226 — [doi:10.2307/2332226](https://doi.org/10.2307/2332226) ↗
  40. Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A conditional transformer language model for controllable generation. ArXiv preprint, abs/1909.05858v2.
  41. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199–22213. Curran Associates, Inc.
  42. Boyang Li, Stephen Lee-Urban, George Johnston, and Mark Riedl. 2013. Story generation with crowdsourced plot graphs. Proceedings of the AAAI Conference on Artificial Intelligence, 27(1):598–604. https://doi.org/10 .1609/aaai.v27i1.8649 find
  43. Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and ‘‘Teknium’’. 2023. OpenOrca: An open dataset of GPT augmented FLAN reasoning traces. find
  44. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  45. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. ArXiv preprint, abs/1907.11692v1.
  46. Vincenzo Lombardo and Rossana Damiano. 2012. Storytelling on mobile devices for cultural heritage. New Review of Hypermedia and Multimedia, 18(1-2):11–35. https://doi.org/10.1080/13614568.2012.617846 — [doi:10.1080/13614568.2012.617846](https://doi.org/10.1080/13614568.2012.617846) ↗
  47. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62–90, Florence, Italy. Asso…
  48. Inbal Magar and Roy Schwartz. 2022. Data contamination: From memorization to exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 157–165, Dublin, Ireland. Association for Computational Linguistics. https://doi.org/1… find
  49. Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2024. Dissociating language and thought in large language models. Trends in Cognitive Sciences, 28(6):517–540. https://doi.org/10.1016/j.tics .2024.01.011, PubMed: 38508911 ↗
  50. Allyssa McCabe and Carole Peterson. 1984. What makes a good story. Journal of Psycholinguistic Research, 13(6):457–480. https://doi.org/10.1007/BF01068179 — [doi:10.1007/bf01068179](https://doi.org/10.1007/bf01068179) ↗
  51. Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. 2018. UMAP: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861. https://doi.org/10.21105/joss .00861 ↗
  52. Blakeley B. McShane, David Gal, Andrew Gelman, Christian Robert, and Jennifer L. find
  53. Tackett. 2019. Abandon statistical significance. The American Statistician, 73(sup1): 235–245. https://doi.org/10.1080/00031305.2018.1527253 ↗
  54. Sara Miller and Lisa Pennycuff. 2008. The power of story: Using storytelling to improve literacy learning. Journal of Cross-Disciplinary Perspectives in Education, 1(1):36–43. find
  55. Jihyung Moon. 2019. Significance test of increase in correlation for NLP evaluations in Python. find
  56. Stefanie Muff, Erlend B. Nilsen, Robert B. O’Hara, and Chloé R. Nater. 2022. Rewriting results sections in the language of evidence. Trends in Ecology & Evolution, 37(3):203–210. https://doi.org/10.1016/j.tree .2021.10.009, PubMed: 34799145 — [doi:10.1016/j.tree](https://doi.org/10.1016/j.tree) ↗
  57. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Progressive learning from complex explanation traces of GPT-4. arXiv preprint arXiv:2306.02707v1.
  58. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241–2252, Copenhagen, Denmark. Association for Computational Linguistics. h…
  59. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Tr… — Training language models to follow instructions with human feedback
  60. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  61. Karl Pearson. 1895. VII. Note on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London, 58(347–352):240–242. https://doi.org/10.1098/rspl.1895.0041 — [doi:10.1098/rspl.1895.0041](https://doi.org/10.1098/rspl.1895.0041) ↗
  62. Maja Popović. 2015. chrF: Character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. https://doi.org/10.18653/v1/W15-3049 ↗
  63. Muhammad Aasim Qureshi, Muhammad Asif, Mohd Fadzil Hassan, Ghulam Mustafa, Muhammad Khurram Ehsan, Aasim Ali, and Unaza Sajid. 2022. A novel auto-annotation technique for aspect level sentiment analysis. Computers, Materials and Continua, 70(3):4987–5004. https://doi.org/10 .32604/cmc.2022.020544 find
  64. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  65. Hannah Rashkin, Asli Celikyilmaz, Yejin Choi, and Jianfeng Gao. 2020. PlotMachines: Outline-conditioned generation with dynamic plot state tracking. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4274–4295, Online. Association for Computation… find
  66. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  67. Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–7. https://doi.org/10.1145/3411763.3451760 ↗
  68. Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for sequence generation tasks. Transactions of the Association for Computational Linguistics, 8:264–280. https://doi.org/10.1162/tacl a 00313 ↗
  69. Stephen Rowcliffe. 2004. Storytelling in science. School Science Review, 86(314):121. find
  70. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7–11, 2024. OpenRevi… find
  71. Charles Spearman. 1961. The proof and measurement of association between two things. In J. J. Jenkins and D. G. Paterson, editors, Studies in Individual Differences: The Search for Intelligence, pages 45–58. Appleton-Century-Crofts. https://doi.org/10.1037/11491 -005 — [doi:10.1037/11491](https://doi.org/10.1037/11491) ↗
  72. James H. Steiger. 1980. Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87(2):245–251. https://doi.org/10.1037/0033-2909.87.2.245 — [doi:10.1037/0033-2909.87.2.245](https://doi.org/10.1037/0033-2909.87.2.245) ↗
  73. Stanley S. Stevens. 1971. Issues in psychophysical measurement. Psychological Review, 78(5):426–450. https://doi.org/10.1037/h0031324 ↗
  74. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. LaMDA: Language models for dialog applications.ArXiv preprint, abs/2201.08239v3.
  75. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971v1.
  76. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  77. Scott R. Turner. 2014. The Creative Process: A Computer Model of Storytelling and Creativity. Psychology Press. find
  78. Raphael Vallat. 2018. Pingouin: Statistics in Python. Journal of Open Source Software, 3(31):1026. https://doi.org/10 .21105/joss.01026. find
  79. Oleg Vasilyev, Vedant Dharnidharka, and John Bohannon. 2020. Fill in the BLANC: Human-free quality estimation of document summaries. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 11–20, Online. Association for Computational Linguistics. https://doi .org/10.1…
  80. Michael Vauth, Hans Ole Hatzel, Evelyn Gius, and Chris Biemann. 2021. Automated event annotation in literary texts. In Proceedings of the Conference on Computational Humanities Research, CHR2021, Amsterdam, The Netherlands, November 17–19, 2021, volume 2989 of CEUR Workshop Proceedings, pages 333–34… find
  81. Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce labeling cost? GPT-3 can help. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4195–4205, Punta Cana, Dominican Republic. Association for Computational Linguistics. https://doi.… ↗
  82. Ronald L. Wasserstein, Allen L. Schirm, and Nicole A. Lazar. 2019. Moving to a world beyond ‘‘p \< 0.05’’. The American Statistician, 73(sup1):1–19. https://doi.org/10.1080/00031305.2019.1583913 ↗
  83. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. Emergent abilities of large language models.Transactions on Ma…
  84. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, Denny Zhou, et al. 2022b. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
  85. Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with ChatGPT. ArXiv preprint, abs/2302.11382v1.
  86. Evan J. Williams. 1959. Regression Analysis, New York. Wiley. find
  87. David Wilmot and Frank Keller. 2021. A temporal variational model for story generation. ArXiv preprint, abs/2109.06807v1. — [arXiv:2109.06807](https://arxiv.org/abs/2109.06807) find
  88. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… — Transformers: State-of-the-Art Natural Language Processing
  89. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020 .emnlp-demos.6 — [doi:10.18653/v1/2020](https://doi.org/10.18653/v1/2020) ↗
  90. Nahathai Wongpakaran, Tinakon Wongpakaran, Danny Wedding, and Kilem L. Gwet. 2013. A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples. BMC Medical Research Methodology, 13:16. https://doi.org/10 .118… find
  91. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, Ne…
  92. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTScore: Evaluating generated text as text generation. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6–14, 2021, virtual, pages 27263–27277.
  93. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26–30, 2020. OpenReview.net. find
  94. Ying Zhang, Stephan Vogel, and Alex Waibel. 2004. Interpreting BLEU/NIST scores: How much improvement do we need to have a better system? In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04). Lisbon, Portugal. European Language Resources Association (E… find
  95. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18–24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Lea…
  96. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi. 2023a. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representat…
  97. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023b. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1–5, 2023. OpenReview.net.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chhun-2024-do-language-models-enjoy-their/