Elamur Research⌕
☰
home / papers / 2024 / WorryWords: Norms of Anxiety Association for over 44k English Words

WorryWords: Norms of Anxiety Association for over 44k English Words

Paper · 2024 · Saif M. Mohammad

Summary

Anxiety, a forward-looking unease about possible bad outcomes, matters for psychology and public health, yet there was no large lexicon linking English words to it. Mohammad builds WorryWords, a manually annotated resource that gives a real-valued anxiety score to 44,450 English words. The vocabulary comes from the NRC Emotion Lexicon, the NRC VAD Lexicon and the Prevalence dataset. Crowd workers on Mechanical Turk rated each word on a bipolar scale running from strong calmness (−3) to strong anxiety (+3). Quality was protected with hidden and feedback-giving gold items, and outlier annotators were removed before aggregation. The scores show high split-half reliability, comparable to established emotion lexicons. The paper uses the lexicon in three ways: to relate anxiety to other affect dimensions and emotions, to examine how children acquire anxiety-related and calm words with age, and to show that simple word counting tracks anxiety change in text streams. The resource is free and comes with translations into more than 100 languages. It supports research in NLP, psychology, social science and health, within stated ethical limits.

Key points

  • WorryWords covers 44,450 English unigrams. It contains 375,796 annotations from 1,020 annotators, mostly in the US and India, with some in the UK and Canada. The mean number of annotations per item is 8.5.
  • Split-half reliability (Spearman ρ) is 0.82, which is in line with existing valence, arousal, dominance and emotion lexicons.
  • The rating scale is a single calm–anxious continuum from −3 to +3. Annotators received a list of near-synonyms for both poles and were told to judge the predominant sense of each word.
  • About 2% of items were gold questions. Half were "popup gold", which gave immediate feedback on errors. The other half were "no-popup gold", which silently monitored annotators in case answers were being shared.
  • For evaluation, the author built 1,000 synthetic anxiety arcs from a Reddit stress dataset (Rastogi et al., 2022). A follow-up resource adds norms for over 10K phrases.

Interesting ideas

  • I1. The paper rates one bipolar axis from calm to anxious instead of marking anxiety as simply present or absent, so neutral and soothing words get distinct values.
  • I2. Gold checks are split into two roles: some items give feedback to train annotators, while others stay hidden to detect gaming or answer-sharing.
  • I3. Annotators were asked what most English speakers would agree with, not how they personally feel. This turns individual opinion into an estimate of a shared norm.
  • I4. The author had no natural gold data for anxiety over time, so he assembled text streams from labeled posts to produce arcs with known ground truth.
  • I5. The ethics guidance limits claims to aggregate, comparative statements about word usage. It warns against judging a single sentence from its words' scores and notes that each score reflects only the dominant sense.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Character and scene states in the world model could carry a signed tension value instead of a boolean "worried" flag. This would let fables show tension rising and resolving.
  • I2 (learn): LLM teacher tiers and witness parsers could be audited the same way. Some known-answer items would be shown as corrections, and others kept as silent negative controls to catch teachers that drift or overfit.
  • I3 (learn): Prompts to LLM teachers for lexical values and attitudes could ask for the consensus association. This should give steadier silver labels for the compiled value lexicons.
  • I4 (learn): Mova could build evaluation stories by concatenating episodes whose states are already known. That would let it check whether the world model's tracked state trajectory matches the intended one.
  • I5 (world): Lexicon scores should enter the world only as weak priors attached to WordNet senses. Frame-level evidence should override them, so that answers never rest on the word scores alone.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️National Research Council CanadaOttawa · Canada59 works · 21,677 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSocial Psychology · 306 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Saif M. Mohammad. 2024. WorryWords: Norms of Anxiety Association for over 44k English Words. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.emnlp-main.910

@inproceedings{mohammad2024worrywords,
  title = {{WorryWords: Norms of Anxiety Association for over 44k English Words}},
  author = {Saif M. Mohammad},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.910},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.910}
}

References

44 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lisa Feldman Barrett. 2017. The theory of constructed emotion: an active inference account of interoception and categorization. Social cognitive and affective neuroscience, 12(1):1–23. ↗
  2. Melissa Bateson, Ben Brilot, and Daniel Nettle. 2011. Anxiety: an evolutionary approach. The Canadian Journal of Psychiatry, 56(12):707–715. find
  3. Katja Beesdo, Susanne Knappe, and Daniel S Pine. 2009. Anxiety and anxiety disorders in children and adolescents: developmental issues and implications for dsm-v. Psychiatric Clinics, 32(3):483–524. find
  4. CR Blease. 2015. Too many ‘friends,’too few ‘likes’? evolutionary psychology and ‘facebook depression’. Review of General Psychology, 19(1):1–13. find
  5. Ruth Breeze. 2020. Angry tweets: A corpus-assisted study of anger in populist political discourse. Journal of Language Aggression and Conflict, 8(1):118–145. find
  6. Peter R Breggin. 2015. The biological evolution of guilt, shame and anxiety: A new theory of negative legacy emotions. Medical hypotheses, 85(1):17–24. ↗
  7. Marc Brysbaert, Paweł Mandera, Samantha F Mc-Cormick, and Emmanuel Keuleers. 2019. Word prevalence norms for 62,000 english lemmas. Behavior research methods, 51:467–479. ↗
  8. Susan Clayton. 2020. Climate anxiety: Psychological responses to climate change. Journal of anxiety disorders, 74:102263. ↗
  9. Paul Ekman. 1992. An argument for basic emotions. Cognition and Emotion, 6(3):169–200. ↗
  10. Morgan R Frank, Lewis Mitchell, Peter Sheridan Dodds, and Christopher M Danforth. 2013. Happiness and the patterns of life: A study of geolocated tweets. Scientific reports, 3(1):2625. ↗
  11. Ned H Kalin. 2020. The critical relationship between anxiety and depression. ↗
  12. Jean Kim and Jack Gorman. 2005. The psychobiology of anxiety. Clinical Neuroscience Research, 4(5-6):335–347. ↗
  13. Victor Kuperman, Hans Stadthagen-Gonzalez, and Marc Brysbaert. 2012. Age-of-acquisition ratings for 30,000 english words. Behavior research methods, 44:978–990. ↗
  14. Jeroen S Legerstee, Nadia Garnefski, Francine C Jellesma, Frank C Verhulst, and Elisabeth MWJ Utens. 2010. Cognitive coping and childhood anxiety disorders. European child & adolescent psychiatry, 19:143–150. ↗
  15. Sandra Mitrović, Oscar William Lithgow-Serrano, and Carlo Schillaci. 2024. Comparing panic and anxiety on a dataset collected from social media. In Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024), pages 153–165, St. Julians, Malta. Association for Computational Linguistics. find
  16. Saif Mohammad. 2011. From once upon a time to happily ever after: Tracking emotions in novels and fairy tales. In Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 105–114, Portland, OR, USA. Association for Computational Linguistics. find
  17. Saif Mohammad. 2018a. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 English words. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 174–184, Melbourne, Australia. Association for Computational Linguistics.
  18. Saif M. Mohammad. 2018b. Word affect intensities. In Proceedings of the 11th Edition of the Language Resources and Evaluation Conference (LREC-2018), Miyazaki, Japan.
  19. Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a word–emotion association lexicon. Computational Intelligence, 29(3):436–465. ↗
  20. Agnes Moors, Jan De Houwer, Dirk Hermans, Sabine Wanmaker, Kevin Van Schie, Anne-Laura Van Harmelen, Maarten De Schryver, Jeffrey De Winne, and Marc Brysbaert. 2013. Norms of valence, arousal, dominance, and age of acquisition for 4,300 dutch words. Behavior research methods, 45(1):169–177. ↗
  21. Emily Ohman, Yuri Bizzoni, Pascale Feldkamp Moreira, and Kristoffer Nielbo. 2024. EmotionArcs: Emotion arcs for 9,000 literary texts. In Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2024), pages 51–66, St. Julians, Malta. Association for Computational Linguistics. find
  22. C.E. Osgood, Suci G., and P. Tannenbaum. 1957. The measurement of meaning. University of Illinois Press. find
  23. Jane Parish. 2000. The age of anxiety. The Sociological Review, 48(2_suppl):1–16. ↗
  24. Robert Plutchik. 1980. A general psychoevolutionary theory of emotion. Emotion: Theory, research, and experience, 1(3):3–33. ↗
  25. John Scott Price. 2013. An evolutionary perspective on anxiety and anxiety disorders. New insights into anxiety disorders, 3:20. ↗
  26. Saif M. Mohammad. 2022. Ethics sheet for automatic emotion recognition and sentiment analysis. Computational Linguistics, 48(2):239–278. ↗
  27. Emily Qureshi-Hurst. 2022. Anxiety, alienation, and estrangement in the context of social media. Religious Studies, 58(3):522–533. ↗
  28. Saif M. Mohammad. 2023. Best practices in the creation and use of emotion lexicons. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Dubrovnik, Croatia. Association for Computational Linguistics. ↗
  29. Aryan Rastogi, Liu Qian, and Erik Cambria. 2022. Stress detection from social media articles: New dataset benchmark and analytical study. In 2022 IEEE World Conference of Computational Intelligence (WCCI). find
  30. Saif M. Mohammad. 2024a. NRC VAD Lexicon v2: Norms for valence, arousal, and dominance for over 55k English terms. In ArXiv. ↗
  31. Saif M. Mohammad. 2024b. Worrywords: A massive lexicon of anxiety associations for English words and phrases. In ArXiv. find
  32. Saif M. Mohammad and Svetlana Kiritchenko. 2015. Using hashtags to capture fine emotion categories from tweets. Computational Intelligence, 31(2):301– 326. ↗
  33. Saif M. Mohammad and Peter D. Turney. 2010. Emotions evoked by common words and phrases: Using Mechanical Turk to create an emotion lexicon. In Proceedings of the NAACL-HLT Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, LA, California.
  34. Jeffrey B Rosen and Jay Schulkin. 1998. From normal fear to pathological anxiety. Psychological review, 105(2):325. ↗
  35. James A Russell. 1980. A circumplex model of affect. Journal of personality and social psychology, 39(6):1161. ↗
  36. Renata Salecl. 2004. On anxiety. Routledge. find
  37. Judy Hanwen Shen and Frank Rudzicz. 2017. Detecting anxiety through Reddit. In Proceedings of the Fourth Workshop on Computational Linguistics and Clinical Psychology — From Linguistic Signal to Clinical Reality, pages 58–65, Vancouver, BC. Association for Computational Linguistics. find
  38. Lisa M Shin and Israel Liberzon. 2010. The neurocircuitry of fear, stress, and anxiety disorders. Neuropsychopharmacology, 35(1):169–191. ↗
  39. All terms in the NRC Emotion Lexicon v1 (Mohammad and Turney, 2013, 2010): It includes about 14,000 common English words with labels indicating whether they are associated with any of the eight notable discrete emotions identified by Plutchik (1980): anger, anticipation, disgust, fear, joy, sadness, surprise, and trust. find
  40. Andy P Siddaway, Peter J Taylor, and Alex M Wood. 2018. Reconceptualizing anxiety as a continuum that ranges from high calmness to high anxiety: The joint importance of reducing distress and increasing well-being. Journal of personality and social psychology, 114(2):e1. ↗
  41. Elizabeth C Stade, Lyle Ungar, Johannes C Eichstaedt, Garrick Sherman, and Ayelet Meron Ruscio. 2023. Depression and anxiety have distinct and overlapping language patterns: Results from a clinical interview. Journal of psychopathology and clinical science. ↗
  42. All terms in the NRC VAD Lexicon v1 (Mohammad, 2018a): It includes about 20,000 common English terms. Notably, it also includes terms common in X (formerly Twitter). Posts on X include non-standard language such as creatively spelled words (happee), hashtags (#takingastand, #lonely) and conjoined words (loveumom). These were taken from the Twitter Emotion Corpus (Mohammad and Kiritchenko, 2015). find
  43. Daniela Teodorescu and Saif Mohammad. 2023. Evaluating emotion arcs across languages: Bridging the global divide in sentiment analysis. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4124–4137, Singapore. Association for Computational Linguistics. ↗
  44. Elsbeth Turcan and Kathy McKeown. 2019. Dreaddit: A Reddit dataset for stress analysis in social media. In Proceedings of the Tenth International Workshop on Health Text Mining and Information Analysis (LOUHI 2019), pages 97–107, Hong Kong. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mohammad-2024-worrywords-norms-anxiety-association-over/