Elamur Research⌕
☰
home / papers / 2023 / Spelling convention sensitivity in neural language models

Spelling convention sensitivity in neural language models

Paper · 2023 · Elizabeth Nielsen, Christo Kirov and Brian Roark

Summary

The paper asks whether large neural LMs trained on mixed web English keep British or American spelling consistent within one generated string. For example, after seeing labour, does the model prefer organisation over organization? The authors note that spelling is a surface-level dependency. Unlike syntax, it can be measured directly in both models and their training corpora, so model behaviour can be traced back to the data.

Using 1706 US/UK word pairs from the open-source American British English Translator, they first measure consistency in corpora such as OWT, English Wikipedia and the BNC. They then probe T5 and GPT2 with prompt templates that pair consistent and inconsistent words. They also fine-tune T5 on a synthetically debiased slice of the BNC.

The corpora turn out to be largely, but not fully, consistent, and they skew heavily American. Both models prefer consistency only partially. The preference is weaker for British spelling, fades with distance, and does not carry over to nonce words. The practical upshot is that systems needing locale-consistent output should post-process.

Key points

  • The probe vocabulary is 1706 American/British pairs taken from data/american_spellings.json of the American British English Translator. It covers -or/-our, -ize/-ise, consonant doubling, -er/-re and one-off items like aluminum/aluminium.
  • Web training data is at least as consistent as curated corpora such as the BNC. Wikipedia and the BNC show relatively high mismatch, explained by many editors and diverse sources.
  • T5 prefers consistent spelling, but more weakly for UK than US, and not robustly for nonce words. Fine-tuning on a synthetically modified BNC portion reduces the American preference.
  • GPT2 prefers consistent continuations in these cases:
Word 1PositionScored to target wordScored to end of sentence
USadjacent94.2%70.1%
USnon-adjacent92.5%67.5%
UKadjacent70.8%49.4%
UKnon-adjacent54.6%45.1%

For UK, this is essentially chance beyond the immediate neighbourhood of the priming word.

  • In OWT itself, a UK word is followed by a UK word with conditional probability 0.66 (adjacent) and 0.72 (non-adjacent). So the data is more consistent than the model, leaving room for improvement.

Interesting ideas

  • I1. Measure the target dependency in the training corpus first, and use that rate as the reference the model should reach; this separates data limits from model limits.
  • I2. Use nonce words that follow a familiar pattern but were never seen, to tell memorised items from learned rules.
  • I3. Treat spelling convention as a document-level property, not a per-word choice, since local evidence fades quickly with distance.
  • I4. Average scores over several prompt templates to dampen sensitivity to wording.
  • I5. Score up to the target word and also to the end of the sentence; the gap reveals purely local behaviour.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Before blaming a parser, check how consistently the silver data agreed by witness parsers itself encodes a given relation or convention, and use that as the ceiling in evaluation reports.
  • I2 (learn): Add invented words with regular English morphology as a negative control, to verify that tagger and lemma rules generalise rather than recall a lexicon.
  • I3 (tokens): Map US/UK variants to one lemma while recording a document-level dialect feature, so later stages see colour and color as the same concept.
  • I4 (learn): When the LLM teacher tiers label data, query with several phrasings and aggregate, flagging items whose label flips across templates.
  • I5 (back): In generation and the compressor, enforce one spelling convention from the meaning graph's document settings, rather than trusting a model that is only locally consistent.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Elizabeth Nielsen, Christo Kirov and Brian Roark. 2023. Spelling convention sensitivity in neural language models. Findings of the Association for Computational Linguistics: EACL 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-eacl.98

@inproceedings{nielsen2023spelling,
  title = {{Spelling convention sensitivity in neural language models}},
  author = {Elizabeth Nielsen, Christo Kirov and Brian Roark},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-eacl.98},
  url = {https://doi.org/10.18653/v1/2023.findings-eacl.98}
}

References

23 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kristian Berg and Mark Aronoff. 2017. Self-organization in the spelling of English suffixes: The emergence of culture out of anarchy. Language, 93(1):37–64. ↗
  2. BNC Consortium. 2007. The British National Corpus, XML edition. Oxford Text Archive, http://www.natcorp.ox.ac.uk. ↗
  3. Claire Bowern. 2015. Linguistic fieldwork : a practical guide, 2nd edition. edition. Palgave Macmillian, Basingstoke, Hampshire. find
  4. Jonathan R. Brennan, Chris Dyer, Adhiguna Kuncoro, and John T. Hale. 2020. Localizing syntactic predictions using recurrent neural network grammars. Neuropsychologia, 146:107479. ↗
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
  6. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005. ↗
  7. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? An analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computational Linguistics. ↗
  8. Kevin Clark and Christopher D. Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. In Empirical Methods on Natural Language Processing.
  9. Stefan L. Frank, Thijs Trompenaars, and Shravan Vasishth. 2016. Cross-linguistic differences in processing double-embedded relative clauses: Working-memory constraints or language statistics? Cognitive Science, 40(3):554–578. ↗
  10. Richard Futrell, Ethan Wilcox, Takashi Morita, and Roger Levy. 2018. RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency. arXiv preprint arXiv:1809.01329. ↗
  11. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1195–1205, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  12. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics. Samuel Johnson. 1755. A Dictionary of the English Language. J. & P. Knapton, London.
  13. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5803–5808, Hong Kong, China. Association for Computational Linguistics. ↗
  14. Adam Kilgarriff and Irene Renau. 2013. esTenTen, a vast web corpus of Peninsular and American Spanish. Procedia - Social and Behavioral Sciences, 95:12–19. Corpus Resources for Descriptive and Applied Studies. Current Challenges and Future Directions: Selected Papers from the 5th International Conference on Corpus Linguistics (CILC2013). find
  15. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  16. Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019. Open sesame: Getting inside BERT’s linguistic knowledge. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 241–253, Florence, Italy. Association for Computational Linguistics.
  17. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521– 535. ↗
  18. Robert C. Moore. 2004. On log-likelihood-ratios and the significance of rare events. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 333–340, Barcelona, Spain. Association for Computational Linguistics. find
  19. Dong Nguyen and Jack Grieve. 2020. Do word embeddings capture spelling variation? In Proceedings of the 28th International Conference on Computational Linguistics, pages 870–881, Barcelona, Spain (Online). International Committee on Computational Linguistics. ↗
  20. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–14, New Orleans, Louisiana. Association for Computational Linguistics. find
  21. Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online. Association for Computational Linguistics. ↗
  22. Albert Webson and Ellie Pavlick. 2022. Do prompt-based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2300–2344, Seattle, United States. Association for Computational Linguistics. ↗
  23. Noah Webster. 1828. An American Dictionary of the English Language. S. Converse, New York. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nielsen-2023-spelling-convention-sensitivity-neural-language/