Elamur Research⌕
☰
home / papers / 2026 / Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Models

Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Models

Summary

The paper examines how well large language models follow strict letter-level rules during generation, a setting where fluent-sounding output is not enough. The test material is 58 daily New York Times Spelling Bee puzzles from summer 2025. Given seven letters, a model must list English words of at least four letters that always contain a designated center letter. The authors run 39 zero-shot configurations from Qwen3 (4B–32B, dense and MoE), Claude Haiku 4.5 and GPT-5-mini, varying reasoning mode and token budget. They compare the outputs with per-word success rates from 10,000 human players per puzzle. Model family matters far more than size. The strongest family roughly doubles the weakest one's F1, mainly through better recall, whereas scaling Qwen3 eightfold yields an 83% gain. Extra thinking budget pays off only for high-capacity models. Model and human difficulty correlate only modestly, and models consistently overlook familiar words with unusual spelling. The released dataset and the diagnosis (models prefer plausible-looking strings over constraint-valid ones) are relevant to any controlled-generation work, though only English was studied.

Key points

  • Data and scale: 58 NYT Spelling Bee puzzles (June 2–July 29, 2025) with 2,710 target word instances (2,007 unique words). The setup has 39 configurations × 58 puzzles, giving 2,262 runs. Human difficulty is per-word solver success (3–97%), possibly hint-assisted.
  • Cross-family vs. scaling: The cross-family gap is 2.0–2.2× (F1 0.761 vs. 0.343), compared with an 83% gain from Qwen3 4B→32B. A partial-correlation analysis excludes tokenizer design as a within-family confound.
  • Thinking budgets: Strong models gain +0.102 to +0.136 F1 from more budget. Qwen3-14B gets worse as budget grows, and the MoE 30B-A3B (3B active parameters) needs a large budget to compensate.
  • Human alignment and failures: Spearman ρ between model and human difficulty is 0.28–0.42. Words like "data", "loll" and "acai" are solved by 83–91% of humans but missed by models 94–98% of the time. On 7+ letter words, humans reach 69% success while small models fall to 0.3%.
  • Resources: Code and outputs are at GitHub ReDASers/spelling-bee-llm-eval. The dataset is on Hugging Face as redasers/spelling-bee-human-difficulty.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3erhoom72odv

@inproceedings{tuck2026orthographic,
  title = {{Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Models}},
  author = {Tuck, Bryan E. and Verma, Rakesh},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3erhoom72odv},
  url = {https://doi.org/10.63317/3erhoom72odv}
}

References

29 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas. 2022. Language models as agent models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5769–5779, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  2. Anthropic. 2025. Claude opus 4 & claude sonnet 4 system card. Technical report, Anthropic. Models: Claude Opus 4, Claude Sonnet 4. find
  3. Lukas Beurer-Kellner et al. 2024. Guiding llms the right way: Fast, non-invasive constrained decoding. arXiv:2403.06988. find
  4. Bastian Bunzeck, Daniel Duran, Leonie Schade, and Sina Zarrieß. 2024. Graphemes vs. phonemes: battling it out in character-based language models. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, pages 54–64, Miami, FL, USA. Association for Computational… find
  5. Yekun Chai, Yewei Fang, Qiwei Peng, and Xuhong Li. 2024. Tokenization falling short: On subword robustness in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1582–1599, Miami, Florida, USA. Association for Computational Linguistics. find
  6. Thao Anh Dang, Limor Raviv, and Lukas Galke. 2025. Tokenization and morphology in multilingual language models: A comparative analysis of mT5 and ByT5. In Proceedings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025), pages 242–257, Southern Denmark Universi… find
  7. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. 2023. Faith and fate: Limits of transformers on composition… find our card
  8. Lukas Edman, Helmut Schmid, and Alexander Fraser. 2024. CUTE: Measuring LLMs’ understanding of their tokens. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3017–3026, Miami, Florida, USA. Association for Computational Linguistics. find
  9. Cristina Garbacea and Qiaozhu Mei. 2025. Why is constrained neural language generation particularly challenging? Transactions on Machine Learning Research. find
  10. Panagiotis Giadikiaroglou, Maria Lymperaiou, Giorgos Filandrianos, and Giorgos Stamou. 2024. Puzzle solving using reasoning of large language models: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11574–11591, Miami, Florida, USA. Associati… find
  11. Chris Hokamp and Qun Liu. 2017. Lexically constrained decoding for sequence generation using grid beam search. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1535–1546, Vancouver, Canada. Association for Computational Linguis… find our card
  12. Inbal Itzhak, Ran Zmigrod, and Ryan Cotterell. 2022. Models in a spelling bee: Language models implicitly learn to spell. In NAACL. find
  13. Abdelhak Kelious, Mathieu Constant, and Christophe Coeur. 2024. Complex word identification: A comparative study between ChatGPT and a dedicated model for this task. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING… find
  14. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. find
  15. Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. 2023. Transformers learn shortcuts to automata. find
  16. Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowan Zellers, Noah A. Smith, and Yejin Choi. 2022. NeuroLogic a*esque decoding: Constrained text generation with lookahead heuristics. In Proceedings of the 2022 Conference of th… find our card
  17. Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2023. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Comput. Surv., 56(2). find our card
  18. Adam Nohejl, Frederikus Hudi, Eunike Andriani Kardinata, Shintaro Ozaki, Maria Angelica Riera Machin, Hongyu Sun, Justin Vasselli, and Taro Watanabe. 2025. Beyond film subtitles: Is YouTube the best approximation of spoken vocabulary? In Proceedings of the 31st International Conference on Computatio… find
  19. OpenAI. 2025. Gpt-5 system card. Technical report, OpenAI. Version: GPT-5 (August 13 2025). find
  20. Kanghee Park, Timothy Zhou, and Loris D’Antoni. 2025. Flexible and efficient grammar-constrained decoding. find
  21. Friedemann Pulvermüller. 2010. Brain-language research: Where is the progress? Biolinguistics. find
  22. Federico Raspanti, Tanir Ozcelebi, and Mike Holenderski. 2025. Grammar-constrained decoding makes large language models better logical parsers. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), pages 485–499, Vienna, Austria. Assoc… find
  23. Soumadeep Saha, Sutanoya Chakraborty, Saptarshi Saha, and Utpal Garain. 2025. Language models are crossword solvers. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pag… find
  24. Ashima Suvarna, Harshita Khandelwal, and Nanyun Peng. 2024. Phonologybench: Evaluating phonological skills of large language models. In ACL Workshop on Knowledge Extraction from LLMs. find
  25. Eshaan Tanwar, Gayatri Oke, and Tanmoy Chakraborty. 2025. Multilingual llms struggle to link orthography and semantics in bilingual word processing. arXiv:2501.09127. find
  26. Bryan E. Tuck and Rakesh Verma. 2025. Unmasking the imposters: How censorship and domain adaptation affect the detection of machine-generated tweets. In Proceedings of the 31st International Conference on Computational Linguistics, pages 9044–9061, Abu Dhabi, UAE. Association for Computational Lingu… find
  27. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… find our card
  28. Kayo Yin and Graham Neubig. 2022. Interpreting language models with contrastive explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 184–198, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  29. Budget Pangram Recall (%) Non-Dict FPs/puzzle Qwen-4B Qwen-8B Qwen-14B Qwen-30B Qwen-32B 16K 4K 4K 8K 8K 1.7 2.7 10.5 0.0 15.8 Claude-Haiku GPT-5-mini 16K 16K 47.3 77.6 Table 5: Pangram recall and non-dictionary false positives (words absent from any standard dictionary) per puzzle at each model’s F… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tuck-2026-orthographic-constraint-satisfaction-human-difficulty/