Elamur Research⌕
☰
home / papers / 2024 / The "LLM World of Words" English free association norms generated by large language models

The "LLM World of Words" English free association norms generated by large language models

Paper · 2024 · Katherine Abramski, Riccardo Improta, Giulio Rossetti et al.

Summary

Researchers who study the conceptual knowledge inside LLMs had no large set of machine-produced free associations that could be compared directly with human norms. The authors built one, called LLM World of Words (LWOW). They took the roughly 12,000 cue words of the human Small World of Words (SWOW) norms and gave each cue to Mistral, Llama3 and Haiku. The prompt imitated the original instructions to human participants, asked for three associates, and was repeated 100 times per cue. All four datasets went through the same preprocessing. Each was then turned into a cognitive network of semantic memory that links cues to their responses. To validate the networks, the authors simulated semantic priming as activation spreading from a prime word. The resulting activation patterns tracked human reaction times from a lexical decision task. The same procedure measured gender stereotypes, which appeared in humans and in all three models with different strengths. The resource lets researchers compare human and LLM associative knowledge directly through behaviour, without relying on embeddings.

Key points

  • The cue list is taken from SWOW-EN.R100.csv: 12,282 cues, each with 100 response sets of 3 words, which gives over 3 million LLM responses per model.
  • The three generating models are Mistral, Llama3 and Haiku. The human SWOW data, after the same extra preprocessing, is called the "Human" dataset. That preprocessing includes lowercasing and removing articles.
  • Validation used 50 targets from a lexical decision task dataset, each paired with a related and an unrelated prime and with mean z-scored reaction times. Activation in the networks was compared against those times.
  • Gender bias was measured with 25 female-related and 25 male-related trait words, taken from a study of gendered job postings. A paired Wilcoxon test compared female-related primes with male-related primes.
  • Effect sizes for female-related targets were:
    • Humans 0.686
    • Haiku 0.669
    • Mistral 0.323
    • Llama3 0.153
  • Spearman correlations between human and LLM activation were about 0.65. The exception was Llama3 on female-related targets, at 0.334.
  • Code and data are available at github.com/LLMWorldOfWords/LWOW.

Interesting ideas

  • I1. Have the model run the same task given to humans, with the same inputs and protocol, so that its output can be scored against existing human norms instead of probing its internals.
  • I2. Repeat an identical prompt 100 times per item to get a response distribution rather than a single answer, which exposes how stable the model's associations are.
  • I3. Validate an association graph by simulating priming as spreading activation and checking it against human reaction times. This is a behavioural test of a knowledge structure that does not need gold annotations.
  • I4. Measure bias with matched prime pairs, comparing a target's activation after a female-related prime with its activation after a male-related prime, and test the paired differences. This turns a vague notion of bias into a controlled contrast.
  • I5. Treat cue-to-response free associations as a graph of explicit, inspectable edges instead of opaque vectors.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): LLM teacher tiers could answer the same items as human-annotated benchmarks, which would show how far each teacher drifts from humans before its silver labels are trusted.
  • I2 (learn): sampling an LLM teacher repeatedly on one sentence and keeping only labels that recur would act as a cheap self-agreement filter next to the multi-parser witnesses.
  • I3 (world): the WordNet/FrameNet-based knowledge graph could be checked by how well activation between word pairs predicts relatedness or priming data, flagging areas where the compiled ontology is weak.
  • I4 (learn): paired-prime contrasts fit naturally into negative-control evaluation, by swapping one entity attribute in a story and checking whether frame or answer outputs shift when they should not.
  • I5 (world): LWOW/SWOW edges could serve as an explainable associative layer under the ontology, for example to rank candidate senses or bridging referents.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TrentoTrento · Italy85 works · 15,524 citations🤝National Research CouncilRome · Italy15 works · 15,649 citations🎓University of PisaPisa · Italy25 works · 3,842 citations🔬Istituto di Scienza e Tecnologie dell'Informazione "Alessandro Faedo"Pisa · Italy4 works · 5,820 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 works

Cite

Cite as Katherine Abramski, Riccardo Improta, Giulio Rossetti and Massimo Stella. 2024. The "LLM World of Words" English free association norms generated by large language models. arXiv:2412.01330.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2412.01330 · arXiv 2412.01330

@misc{abramski2024llm,
  title = {{The "LLM World of Words" English free association norms generated by large language models}},
  author = {Katherine Abramski, Riccardo Improta, Giulio Rossetti and Massimo Stella},
  year = {2024},
  doi = {10.48550/arxiv.2412.01330},
  eprint = {2412.01330},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.01330}
}

References

42 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. De Deyne, S., Navarro, D. J. & Storms, G. Better explanations of lexical and semantic cognition using networks derived from continued rather than single-word associations. Behav. research methods 45, 480–498 (2013). find
  2. Kenett, Y. N., Levi, E., Anaki, D. & Faust, M. The semantic distance task: Quantifying semantic distance with semantic network path length. J. Exp. Psychol. Learn. Mem. Cogn. 43, 1470 (2017). find
  3. Aitchison, J. Words in the mind: An introduction to the mental lexicon (John Wiley & Sons, 2012). find
  4. Vankrunkelsven, H., Verheyen, S., Storms, G. & De Deyne, S. Predicting lexical norms: A comparison between a word association model and text-based word co-occurrence models. J. cognition 1 (2018). find
  5. Samuel, G., Stella, M., Beaty, R. E. & Kenett, Y. N. Predicting openness to experience via a multiplex cognitive network approach. J. Res. Pers. 104, 104369 (2023). find
  6. Shiffrin, R. & Mitchell, M. Probing the psychology of ai models. Proc. Natl. Acad. Sci. 120, e2300963120 (2023). find
  7. Binz, M. & Schulz, E. Using cognitive psychology to understand gpt-3. Proc. Natl. Acad. Sci. 120, e2218523120 (2023). find
  8. Steyvers, M. & Tenenbaum, J. B. The large-scale structure of semantic networks: Statistical analyses and a model of semantic growth. Cogn. science 29, 41–78 (2005). find
  9. Stella, M., De Nigris, S., Aloric, A. & Siew, C. S. Forma mentis networks quantify crucial differences in stem perception between students and experts. PloS one 14, e0222870 (2019). find
  10. De Deyne, S. & Storms, G. Word associations: Network and semantic properties. Behav. research methods 40, 213–231 (2008). find
  11. De Deyne, S., Navarro, D. J., Perfors, A., Brysbaert, M. & Storms, G. The “small world of words” english word association norms for over 12,000 cue words. Behav. research methods 51, 987–1006 (2019). find
  12. Hill, F., Korhonen, A. & Bentz, C. A quantitative empirical analysis of the abstract/concrete distinction. Cogn. science 38, 162–177 (2014). find
  13. Schnabel, K. & Asendorpf, J. B. Free associations as a measure of stable implicit attitudes. Eur. J. Pers. 27, 39–50 (2013). find
  14. Citraro, S., Vitevitch, M. S., Stella, M. & Rossetti, G. Feature-rich multiplex lexical networks reveal mental strategies of early language learning. Sci. Reports 13, 1474 (2023). find
  15. Kenett, Y. N. & Austerweil, J. L. Examining search processes in low and high creative individuals with random walks. In CogSci, vol. 8, 313–318 (2016). find
  16. Beaty, R. E. & Kenett, Y. N. Associative thinking at the core of creativity. Trends Cogn. Sci. (2023). find
  17. Benedek, M. et al. How semantic memory structure and intelligence contribute to creative thought: A network science approach. Think. & Reason. 23, 158–183 (2017). find
  18. Kenett, Y. N., Gold, R. & Faust, M. The hyper-modular associative mind: a computational analysis of associative responses of persons with asperger syndrome. Lang. Speech 59, 297–317 (2016). find
  19. Apidianaki, M. From word types to tokens and back: A survey of approaches to word meaning representation and interpretation. Comput. Linguist. 1–60 (2022).
  20. Rodriguez, M. A. & Merlo, P. Word associations and the distance properties of context-aware word embeddings. In Proceedings of the 24th Conference on Computational Natural Language Learning, 376–385 (2020). find
  21. Yao, P., Renwick, T. & Barbosa, D. Wordties: Measuring word associations in language models via constrained sampling. In Findings of the Association for Computational Linguistics: EMNLP 2022, 5959–5970 (2022). find
  22. Bolukbasi, T., Chang, K.-W., Zou, J. Y., Saligrama, V. & Kalai, A. T. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Adv. neural information processing systems 29 (2016).
  23. Kurita, K., Vyas, N., Pareek, A., Black, A. W. & Tsvetkov, Y. Measuring bias in contextualized word representations. arXiv preprint arXiv:1906.07337 (2019). ↗
  24. Manzini, T., Lim, Y. C., Tsvetkov, Y. & Black, A. W. Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings. arXiv preprint arXiv:1904.04047 (2019). ↗
  25. Caliskan, A., Bryson, J. J. & Narayanan, A. Semantics derived automatically from language corpora contain human-like biases. Science 356, 183–186 (2017). find
  26. Bommasani, R., Davis, K. & Cardie, C. Interpreting pretrained contextualized representations via reductions to static embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4758–4781 (2020).
  27. Greenwald, A. G., McGhee, D. E. & Schwartz, J. L. Measuring individual differences in implicit cognition: the implicit association test. J. personality social psychology 74, 1464 (1998). find
  28. Srivastava, A. et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 (2022). ↗
  29. Hagendorff, T. Machine psychology: Investigating emergent capabilities and behavior in large language models using psychological methods. arXiv preprint arXiv:2303.13988 (2023). ↗
  30. Wang, Y. et al. The fluency-based semantic network of llms differs from humans. Comput. Hum. Behav. Artif. Humans 100103 (2024). find
  31. Suresh, S. et al. Conceptual structure coheres in human cognition but not in large language models. arXiv preprint arXiv:2304.02754 (2023). ↗
  32. Digutsch, J. & Kosinski, M. Overlap in meaning is a stronger predictor of semantic activation in gpt-3 than in humans. Sci. Reports 13, 5035 (2023). find
  33. Abramski, K., Citraro, S., Lombardi, L., Rossetti, G. & Stella, M. Cognitive network science reveals bias in gpt-3, gpt-3.5 turbo, and gpt-4 mirroring math anxiety in high-school students. Big Data Cogn. Comput. 7, 124 (2023). find
  34. Nelson, D. L., McEvoy, C. L. & Schreiber, T. A. The university of south florida free association, rhyme, and word fragment norms. Behav. Res. Methods, Instruments, & Comput. 36, 402–407 (2004). find
  35. Wilson, M., Kiss, G. et al. Eat: The edinburgh associative corpus. Oxf. Text Arch. Core Collect. (1988). — WordNet: A Lexical Database for English
  36. Siew, C. S. spreadr: An r package to simulate spreading activation in a network. Behav. Res. Methods 51, 910–929 (2019). find
  37. Hutchison, K. A. et al. The semantic priming project. Behav. research methods 45, 1099–1114 (2013). find
  38. Collins, A. M. & Loftus, E. F. A spreading-activation theory of semantic processing. Psychol. review 82, 407 (1975). find
  39. Neely, J. H. Semantic priming effects in visual word recognition: A selective review of current findings and theories. Basic processes reading 264–336 (2012). find
  40. McNamara, T. P. Semantic priming: Perspectives from memory and word recognition (Psychology Press, 2005). find
  41. Collins, A. M. & Quillian, M. R. Retrieval time from semantic memory. J. verbal learning verbal behavior 8, 240–247 (1969). find
  42. Gaucher, D., Friesen, J. & Kay, A. C. Evidence that gendered wording in job advertisements exists and sustains gender inequality. J. personality social psychology 101, 109 (2011). Author contributions statement K.A., G.R. and M.S. conceived the experiments, K.A., R.I. and G.R. conducted the experiments, K.A. analyzed the results and visualized the data. All authors reviewed the manuscript. Competing interests The authors declare no competing interests. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abramski-2024-llm-world-words-english-free/