Elamur Research⌕
☰
home / papers / 2026 / Finding Sense in Nonsense with Generated Contexts: Perspectives from Humans and Language Models

Finding Sense in Nonsense with Generated Contexts: Perspectives from Humans and Language Models

Summary

Benchmarks of semantically deviant sentences are widely used to probe semantic interpretation, and acceptability filtering is used in data cleaning and hallucination detection. These uses often treat odd but interpretable sentences the same as true gibberish. The authors separate these two cases: "anomalous" sentences become meaningful once a suitable situation is supplied, while "pure nonsense" stays uninterpretable in any context.

They sampled 40 sentences from each of five datasets (ADEPT, BLiMP, PAP, CConS, Cusp). Phi 4 Mini Instruct and Llama 3.1 8B Instruct each wrote a short context in which a sentence could make sense. Humans and both LLMs then rated sensicality on a 7-point Likert scale, both with and without these contexts. The authors also categorized the generated contexts by theme.

Most sentences were judged at worst anomalous, and only a small share was genuinely nonsensical. Many sentences were judged sensible even without any context. Both LLMs proved capable of producing contexts that make sentences more sensible. The findings suggest that existing "nonsense" datasets overstate how much meaningless material they contain. They also suggest that context-free acceptability filtering may wrongly discard unusual but valid data.

Key points

  • The sample covers 200 sentences, 40 per dataset, with 20 drawn per subtype where a dataset has subtypes. Examples of subtypes are BLiMP animate subject passive/trans, PAP abstract/concrete, and CConS explicit/implicit.
  • Each LLM rated every sentence six times: two prompt wordings × (no context, its own context, the other model's context). Agreement between human annotators reached Kendall's τ ≈ 0.164 (p < 0.001), which is significant but noisy.
  • Generated contexts raised sentence sensicality 72.7% of the time for Phi and 76.4% for Llama.
  • The most frequent context theme was "Specific Setting," followed by Fantasy. The other themes were Figurative, Story, Humorous, Art, Children, and the rare Jargon.
  • Themes clustered more strongly by dataset than by model:
    • CConS accounted for over half of the Art and Children contexts.
    • Cusp, which was derived from metaphors, had a rate of figurative explanations between two and five times that of any other dataset.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.starsem-conference.6

@inproceedings{olsen2026finding,
  title = {{Finding Sense in Nonsense with Generated Contexts: Perspectives from Humans and Language Models}},
  author = {Olsen, Katrina and Padó, Sebastian},
  booktitle = {Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.starsem-conference.6},
  url = {https://doi.org/10.18653/v1/2026.starsem-conference.6}
}

References

32 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi-ling Chen, Qi Dai, Xiyang Dai, Ruchao Fan, and Xiren Zhou. 2025. Phi-4-mini… find our card
  2. Márta Abrusán. 2019. Semantic anomaly, pragmatic infelicity, and ungrammaticality. Annual Review of Linguistics, 5(1):329–351. find
  3. Alan Agresti. 2002. Categorical Data Analysis. John Wiley & Sons. find
  4. Mira Ait-Saada and Mohamed Nadif. 2023. Unsupervised anomaly detection in multi-topic short-text corpora. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1392–1403, Dubrovnik, Croatia. Association for Computational Linguistics. find
  5. Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2013. Concreteness ratings for 40 thousand generally known English word lemmas. Behavior Research Methods, 46(3):904–911. find
  6. Eileen R. Cardillo, Gwenda L. Schmidt, Alexander Kranjec, and Anjan Chatterjee. 2010. Stimulus design is an obstacle course: 560 matched literal and metaphorical sentences for testing neural hypotheses about metaphor. Behavior Research Methods, 42(3):651–664. find
  7. Eileen R. Cardillo, Christine Watson, and Anjan Chatterjee. 2016. Stimulus needs are a moving target: 240 additional matched literal and metaphorical sentences for testing neural hypotheses about metaphor. Behavior Research Methods, 49(2):471–483. find
  8. Annerose Eichel and Sabine Schulte im Walde. 2023. A dataset for physical and abstract plausibility and sources of human disagreement. In Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII), pages 31–45, Toronto, Canada. Association for Computational Linguistics. find
  9. Ali Emami, Ian Porada, Alexandra Olteanu, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Cheung. 2021. ADEPT: An adjective-dependent plausibility task. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natura… find
  10. Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretations difficult. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3719–3728, Brussels, Belgium. Associatio… find
  11. Emily Goodwin, Koustuv Sinha, and Timothy J. O’Donnell. 2020. Probing linguistic systematicity. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1958– 1969, Online. Association for Computational Linguistics. find
  12. Bernadeta Griciūtė, Marc Tanti, and Lucia Donatelli. 2022. On the cusp of comprehensibility: Can language models distinguish between metaphors and nonsense? In Proceedings of the 3rd Workshop on Figurative Language Processing (FLP), pages 173– 177, Abu Dhabi, United Arab Emirates (Hybrid). Associa… find
  13. Ashim Gupta, Giorgi Kvernadze, and Vivek Srikumar. 2021. BERT & family eat word salad: Experiments with text understanding. Proceedings of the AAAI Conference on Artificial Intelligence, 35(14):12946– 12954. find
  14. Matt A. Johnson and Adele E. Goldberg. 2013. Evidence for automatic accessing of constructional meaning: Jabberwocky sentences prime associated verbs. Language and Cognitive Processes, 28(10):1439–1452. find
  15. Kazushi Kondo, Saku Sugawara, and Akiko Aizawa. 2023. Probing physical reasoning with counter-commonsense context. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 603–612, Toronto, Canada. Association for Computational Lingui… find
  16. Jey Han Lau, Carlos Armendariz, Shalom Lappin, Matthew Purver, and Chang Shu. 2020. How furiously can colorless green ideas sleep? Sentence acceptability in context. Transactions of the Association for Computational Linguistics, 8:296–310. find
  17. Bai Li, Zining Zhu, Guillaume Thomas, Yang Xu, and Frank Rudzicz. 2021. How is BERT surprised? Layerwise detection of linguistic anomalies. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Proce… find
  18. Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. WANLI: Worker and AI collaboration for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826–6847, Abu Dhabi, United Arab Emirates. Association for Computat… find our card
  19. John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pa… find our card
  20. David Patterson, Joseph Gonzalez, Urs Hölzle, Quoc Le, Chen Liang, Lluís-Miquel Munguía, Daniel Rothchild, David So, Maud Texier, and Jeffrey Dean. 2022. The carbon footprint of machine learning training will plateau, then shrink. https://arxiv.org/abs/2204.05149. — [arXiv:2204.05149](https://arxiv.org/abs/2204.05149) ↗
  21. Paolo Pedinotti, Eliana Di Palma, Ludovica Cerini, and Alessandro Lenci. 2021. A howling success or a working sea? Testing what BERT knows about metaphors. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 192–204, Punta Cana, Dominican Re… find
  22. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711, Singapore. Association for Computational Linguistics. find
  23. Weiyan Shi, Emily Dinan, Adi Renduchintala, Daniel Fried, Athul Jacob, Zhou Yu, and Mike Lewis. 2023. AutoReply: Detecting nonsense in dialogue with discriminative replies. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 294–309, Singapore. Association for Computation… find
  24. Jiajun Song, Zhuoyan Xu, and Yiqiao Zhong. 2025. Out-of-distribution generalization via composition: A lens through induction heads in transformers. find
  25. Proceedings of the National Academy of Sciences, 122(6):e2417182122. find
  26. Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p… find
  27. Eva Maria Vecchi, Marco Baroni, and Roberto Zamparelli. 2011. (Linear) maps of the impossible: Capturing semantic anomalies in distributional space. In Proceedings of the Workshop on Distributional Semantics and Compositionality, pages 1–9, Portland, Oregon, USA. Association for Computational Lingui… find
  28. Eva Maria Vecchi, Marco Marelli, Roberto Zamparelli, and Marco Baroni. 2017. Spicy adjectives and nominal donkeys: Capturing semantic deviance using compositionality in distributional spaces. Cognitive science, 41 1:102–136. find
  29. Yang Wang, Chenghao Xiao, Chia-Yi Hsiao, Zi Yan Chang, Chi-Li Chen, Tyler Loakman, and Chenghua Lin. 2025. Drivel-ology: Challenging LLMs with interpreting nonsense with depth. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 23074–23096, Suzhou, China… find
  30. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377– 392. find
  31. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. find
  32. Franziska Weeber, Tanise Ceron, and Sebastian Padó. 2026. Do political opinions transfer between western languages? An analysis of unaligned and aligned multilingual LLMs. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long P… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/olsen-2026-finding-sense-nonsense-generated-contexts/