Elamur Research⌕
☰
home / papers / 2026 / Speaking of Language: Reflections on Metalanguage Research in NLP

Speaking of Language: Reflections on Metalanguage Research in NLP

Paper · 2026 · Nathan Schneider and Antonios Anastasopoulos

Summary

This position paper argues that NLP should pay much more attention to metalanguage, now that fluent LLM chatbots are widely used. Metalanguage means language about language. It can be natural, such as statements about grammar or idioms, or symbolic, such as POS tags, bracketing, or logical forms. The authors describe three ways metalanguage appears in LLM work:

  • user instructions that refer to text or communication;
  • description tasks, such as defining words, analyzing grammar, or inducing rules;
  • interpretability studies that compare model internals with linguistic descriptions.

They then review projects from their own labs. At George Mason, Anastasopoulos's group uses grammar books and dictionaries to translate low-resource languages. It also extracts teachable grammar rules from raw corpora. At Georgetown, Schneider's group studies metalinguistic question answering for second-language learners and the use/mention distinction in legal interpretation. The paper presents no new dataset or model. Its contribution is a conceptual framework and a research agenda. The authors argue that fields like education, language documentation, and law depend on reasoning about language itself. This makes metalinguistic ability an important test of LLM robustness.

Key points

  • Table 1 sorts metalinguistic tasks along two axes: symbolic vs. natural, and instance-level vs. system-level. Examples include tagging/parsing, grammaticality questions, grammar rule induction, and writing dictionary definitions.
  • Hus and Anastasopoulos (2024) extended the "Machine Translation from One Book" approach (Tanzer et al., 2024) to 16 languages. Combining dictionary entries with the full grammar book gave chrF++ scores of 25–55 for translating Chuvash, Dogri, and Kalamang into English.
  • Subsequent work by Marmonier et al. (2025) and by Aycock et al. (2025) raises doubts about whether current LLMs actually exploit the grammar content.
  • Grambank, the typological database discussed alongside WALS, covers 2,467 language varieties with 195 grammatical features.
  • The paper proposes 13 open research questions in three groups: intrinsic evaluation, interpretability, and extrinsic uses. Examples include use vs. mention, self-referential sentences, metalinguistic negation, and how well explicit model explanations match the model's implicit linguistic knowledge.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgetown UniversityWashington · United States88 works · 4,732 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Nathan Schneider and Antonios Anastasopoulos. 2026. Speaking of Language: Reflections on Metalanguage Research in NLP. Proceedings of The Big Picture v2: Crafting a Research Narrative.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bigpicture-main.9

@inproceedings{schneider2026speaking,
  title = {{Speaking of Language: Reflections on Metalanguage Research in NLP}},
  author = {Nathan Schneider and Antonios Anastasopoulos},
  booktitle = {Proceedings of The Big Picture v2: Crafting a Research Narrative},
  year = {2026},
  doi = {10.18653/v1/2026.bigpicture-main.9},
  url = {https://doi.org/10.18653/v1/2026.bigpicture-main.9}
}

References

74 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tatsuya Aoyama and Nathan Schneider. 2022. Probeless probing of BERT’s layer-wise linguistic knowledge with masked word prediction. In Proc. of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop,… find
  2. Tjaša Arčon, Matej Klemen, Marko Robnik-Šikonja, and Kaja Dobrovoljc. 2026. Evaluating metalinguistic knowledge in large language models across the world’s languages. arXiv:2602.02182 [cs]. — [arXiv:2602.02182](https://arxiv.org/abs/2602.02182) find
  3. Seth Aycock, David Stap, Di Wu, Christof Monz, and Khalil Simaan. 2025. Can LLMs really learn to translate a low-resource language from one grammar book? In International Conference on Learning Representations, volume 2025, pages 12334–12357. find
  4. Emi Baylor, Esther Ploeger, and Johannes Bjerva. 2023. The past, present, and future of typological databases in NLP. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1163–1169, Singapore. Association for Computational Linguistics. find
  5. Andrew Bean, Simi Hellsten, Harry Mayne, Jabez Magomere, Ethan A., Ryan Chi, Scott A. Hale, and Hannah R. Kirk. 2024. LINGOLY: A benchmark of olympiad-level linguistic reasoning puzzles in low-resource and extinct languages. find
  6. Gašper Beguš, Maksymilian D abkowski, and Ryan Rhodes. 2025. Large linguistic models: investigating LLMs’ metalinguistic abilities. IEEE Transactions on Artificial Intelligence, 6(12):3453–3467. find
  7. Shabnam Behzad, Keisuke Sakaguchi, Nathan Schneider, and Amir Zeldes. 2023. ELQA: A corpus of metalinguistic questions and answers about English. In Proc. of ACL.
  8. Shabnam Behzad, Amir Zeldes, and Nathan Schneider. 2024. To ask LLMs about English grammaticality, prompt them in a different language. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15622–15634, Miami, Florida, USA. Association for Computational Linguistics. find
  9. Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: on meaning, form, and understanding in the age of data. In Proc. of ACL, pages 5185–5198, Online.
  10. Roger Berry. 2005. Making the most of metalanguage. Language Awareness, 14(1):3–20. find
  11. Steven Bird. 2022. Local languages, third spaces, and other high-resource scenarios. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7817–7829, Dublin, Ireland. Association for Computational Linguistics. find
  12. Johannes Bjerva, Elizabeth Salesky, Sabrina J. Mielke, Aditi Chaudhary, Giuseppe G. A. Celano, Edoardo Maria Ponti, Ekaterina Vylomova, Ryan Cotterell, and Isabelle Augenstein. 2020. find
  13. SIGTYP 2020 shared task: Prediction of typological features. In Proceedings of the Second Workshop on Computational Research in Linguistic Typology, pages 1–11, Online. Association for Computational Linguistics. find
  14. Valerie C. Brannon. 2023. Statutory interpretation: theories, tools, and trends. Report R45153, Congressional Research Service. find
  15. Valerie C. Brannon. 2025. Canons of construction: a brief overview. In Focus IF12992, Congressional Research Service. find
  16. Stephen Breyer. 2023. County of Maui v. Hawaii Wildlife Fund. 140 S. Ct. 1462. find
  17. Aditi Chaudhary, Antonios Anastasopoulos, Adithya Pratapa, David R. Mortensen, Zaid Sheikh, Yulia Tsvetkov, and Graham Neubig. 2020. Automatic extraction of rules governing morphological agreement. In Proc. of EMNLP. find
  18. Aditi Chaudhary, Arun Sampath, Ashwin Sheshadri, Antonios Anastasopoulos, and Graham Neubig. 2023. Teacher perception of automatically extracted grammar concepts for L2 language learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3776–3793, Singapore. Associatio… find
  19. Aditi Chaudhary, Kayo Yin, Antonios Anastasopoulos, and Graham Neubig. 2021. When is wall a pared and when a muro?: Extracting rules governing lexical selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6911–6929, Online and Punta Cana, Dominic… find
  20. Kristina Gligorić, Myra Cheng, Lucia Zheng, Esin Durmus, and Dan Jurafsky. 2024. NLP systems that can’t tell use from mention censor counterspeech, but teaching the distinction helps. In Proc. of NAACL-HLT, pages 5942–5959, Mexico City, Mexico. find
  21. Nathan Chi, Teodor Malchev, Riley Kong, Ryan Chi, Lucas Huang, Ethan Chi, R. McCoy, and Dragomir Radev. 2024. ModeLing: a novel dataset for testing linguistic reasoning in language models. In Proc. of the 6th Workshop on Research in Computational Linguistic Typology and Multilingual NLP. find
  22. Laurence R. Horn. 1985. Metalinguistic negation and pragmatic ambiguity. Language, 61(1):121–174. find
  23. Mukund Choudhary, KV Aditya Srivatsa, Gaurja Aeron, Antara Raaghavi Bhattacharya, Dang Khoa Dang Dinh, Ikhlasul Akmal Hanif, Daria Kotova, Ekaterina Kochmar, and Monojit Choudhury. 2025. UNVEIL-ING: What makes linguistics olympiad puzzles tricky for LLMs? In Proc. of COLM. find
  24. Sara Court and Micha Elsner. 2024. Shortcomings of LLMs for low-resource translation: Retrieval and understanding are both the problem. In Proceedings of the Ninth Conference on Machine Translation, pages 1332–1354, Miami, Florida, USA. Association for Computational Linguistics. find
  25. Hal Daumé, III and Lyle Campbell. 2007. A Bayesian model for discovering typological implications. In Proc. of ACL, pages 65–72, Prague, Czech Republic. find
  26. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
  27. William N. Eskridge, Brian G. Slocum, and Kevin Tobia. 2023. Textualism’s defining moment. Columbia Law Review, 123(6):1611–1698. find
  28. Allyson Ettinger, Jena Hwang, Valentina Pyatkin, Chandra Bhagavatula, and Yejin Choi. 2023. “You are an expert linguistic annotator”: Limits of LLMs as analyzers of Abstract Meaning Representation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8250–8263, Singapore. find
  29. Charles J. Fillmore. 1985. Frames and the semantics of understanding. Quaderni di Semantica, 6(2):222– 254. find
  30. Luke Gessler and Nathan Schneider. 2023. Syntactic inductive bias in transformer language models: especially helpful for low-resource languages? In Proc. of CoNLL, pages 238–253, Singapore. find
  31. Michael Ginn and Alexis Palmer. 2025. LLM dependency parsing with in-context rules. In Proceedings of the 1st Joint Workshop on Large Language Models and Structure Modeling (XLLM 2025), pages 186– 196, Vienna, Austria. Association for Computational Linguistics.
  32. Zellig S. Harris. 1951. Methods in Structural Linguistics. University of Chicago Press. find
  33. Kristen Howell, Emily M Bender, Michel Lockwood, Fei Xia, and Olga Zamaraeva. 2017. Inferring case systems from igt: Enriching the enrichment. In Proceedings of the 2nd Workshop on the Use of Computational Methods in the Study of Endangered Languages, pages 67–75. find
  34. Jennifer Hu and Roger Levy. 2023. Prompting is not a substitute for probability measurements in large language models. In Proc. of EMNLP.
  35. Jonathan Hus and Antonios Anastasopoulos. 2024. Back to school: Translation using grammar books. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20207–20219, Miami, Florida, USA. Association for Computational Linguistics. find
  36. Jonathan Hus and Antonios Anastasopoulos. 2026. A rag approach for typological database completion. In Proceedings of the Eighth Workshop on Computational Research in Linguistic Typology, Rabbat, Morocco. Association for Computational Linguistics. find
  37. Michael Kranzlein. 2024. Unpacking Meaning with Natural Language Processing: Legal Metalanguage Analysis and Long-Tail Calibration. Ph.D. dissertation, Georgetown University. find
  38. Michael Kranzlein, Nathan Schneider, and Kevin Tobia. 2024. CuRIAM: Corpus Re Interpretation and Metalanguage in U.S. Supreme Court Opinions. In Proc. of LREC-COLING. find
  39. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Linguistic knowledge and transferability of contextual representations. In Proc. of NAACL-HLT, pages 1073–1094, Minneapolis, Minnesota.
  40. Malik Marmonier, Rachel Bawden, and Benoît Sagot. 2025. Explicit learning and the LLM in machine translation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 31372–31422, Suzhou, China. Association for Computational Linguistics. find
  41. Ndapa Nakashole. 2026. Grammar as control: Modular language generation for the long tail. In Proc. of ACL, San Diego, California.
  42. Kevin Newsom. 2024. Concurring opinion in Snell v. United Specialty Insurance Co. United States Court of Appeals For the Eleventh Circuit, 22-12581. find
  43. Juri Opitz, Shira Wein, and Nathan Schneider. 2025. Natural language processing RELIES on linguistics. Computational Linguistics, 51(3):1009–1032.
  44. Ellie Pavlick. 2023. Symbols and grounding in large language models. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 381(2251):20220041. find
  45. Renhao Pei, Yihong Liu, Peiqin Lin, François Yvon, and Hinrich Schuetze. 2025. Understanding in-context machine translation for low-resource languages: A case study on Manchu. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 87… find
  46. Dawson Petersen, Abhishek Purushothama, and Nathan Schneider. 2026. Sense and sensitivity: “reasoning” models are more robust, but can diverge from human consensus in a legal interpretation task. In Proc. of CoNLL, San Diego, California.
  47. Jakob Prange, Nathan Schneider, and Lingpeng Kong. 2022. Linguistic frameworks go toe-to-toe at neuro-symbolic language modeling. In Proc. of NAACL-HLT, pages 4375–4391, Seattle, United States. find
  48. Abhishek Purushothama, Junghyun Min, Brandon Waldon, and Nathan Schneider. 2026a. Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysis. In Proc. of the Ninth Annual ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT),… — [arXiv:2510.25356](https://arxiv.org/abs/2510.25356) find
  49. Abhishek Purushothama, Emma Thronson, Alexia Guo, and Amir Zeldes. 2026b. Syntax as a Rosetta Stone: Universal Dependencies for in-context Coptic translation. In Findings of ACL. arXiv preprint: 2604.18758 [cs]. find
  50. Josh Rozner, Christopher Potts, and Kyle Mahowald. 2021. Decrypting cryptic crosswords: semantically complex wordplay puzzles as a target for NLP. In Advances in Neural Information Processing Systems, volume 34, pages 11409–11421. find
  51. Gözde Gül Şahin, Yova Kementchedjhieva, Phillip Rust, and Iryna Gurevych. 2020. PuzzLing Machines: a challenge on learning from small data. In Proc. of ACL. find
  52. Siyuan Song, Jennifer Hu, and Kyle Mahowald. 2025. Language models fail to introspect about their knowledge of language. In Proc. of COLM.
  53. Chihiro Taguchi, J Elizabeth Liebl, Antonios Anastasopoulos, David Chiang, and Géraldine Walther. 2025. Digital documentation for diasporic data: challenges, opportunities, and solutions for working with diaspora communities. In 9th International Conference on Language Documentation & Conservation (… find
  54. Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. 2024. A benchmark for learning to translate a new language from one grammar book. In International Conference on Learning Representations, volume 2024, pages 18955–18985. find
  55. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proc. of ACL, pages 4593–4601, Florence, Italy.
  56. Tristan Thrush, Jared Moore, Miguel Monares, Christopher Potts, and Douwe Kiela. 2024. I am a strange dataset: metalinguistic tests for language models. In Proc. of ACL. find
  57. Yuanhe Tian, Fei Xia, and Yan Song. 2024. Large language models are no longer shallow parsers. In Proc. of ACL, pages 7131–7142, Bangkok, Thailand. find
  58. Kevin Tobia. 2022. Experimental jurisprudence. The University of Chicago Law Review, 89(3):735–802. find
  59. Kevin Tobia, Brian G. Slocum, and Victoria Nourse. 2022. Statutory Interpretation from the outside. Columbia Law Review, 122(1):213–330. find
  60. Kevin P. Tobia. 2020. Testing ordinary meaning. Harvard Law Review, 134(2):726–806. find
  61. Eduardo Sánchez, Belen Alastruey, Christophe Ropers, Arina Turkatenko, Pontus Stenetorp, Mikel Artetxe, and Marta R. Costa-jussà. 2025. Linguini: A benchmark for language-agnostic linguistic reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchm… find
  62. Antonin Scalia and Bryan A. Garner. 2012. Reading law: the interpretation of legal texts. Thomson/West, St. Paul, MN. find
  63. Hedvig Skirgård, Hannah J. Haynie, Damián E. Blasi, Harald Hammarström, Jeremy Collins, Jay J. Latarche, Jakob Lesage, Tobias Weber, Alena Witzlack-Makarevich, Sam Passmore, Angela Chira, Luke Maurits, Russell Dinnage, Michael Dunn, Ger Reesink, Ruth Singer, Claire Bowern, Patience Epps, Jane Hill,… find
  64. Brandon Waldon, Cleo Condoravdi, James Pustejovsky, Nathan Schneider, and Kevin Tobia. 2025a. Reading law with linguistics: the statutory interpretation of artifact nouns. Harvard Journal on Legislation, 62(2):415–467. find
  65. Brandon Waldon, Nathan Schneider, Ethan Wilcox, Amir Zeldes, and Kevin Tobia. 2025b. Large language models for legal interpretation? Don’t take their word for it. Georgetown Law Journal, 114(1):115–183. find
  66. Brandon Waldon, Micaela Wells, Devika Tiwari, Meru Gopalan, and Nathan Schneider. 2025c. Legal-CGEL: Analyzing legal text in the CGELBank framework. In Proc. of the 23rd International Workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2025), pages 148–153, Ljubljana, Slovenia. find
  67. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. arXiv:2211.00593 [cs]. find
  68. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics, 8:377– 392.
  69. Shira Wein and Nathan Schneider. 2024. Lost in translationese? Reducing translation effect using Abstract Meaning Representation. In Proc. of EACL, pages 753–765, St. Julian’s, Malta.
  70. Micaela Wells, Brandon Waldon, and Nathan Schneider. 2025. Scope ambiguity resolution of negated connectives in English corpora. In Proc. of the Annual Meeting of the Cognitive Science Society, volume 47, page 6608. find
  71. Shomir Wilson. 2011. A computational theory of the use-mention distinction in natural language. Ph.D. dissertation, University of Maryland, College Park, Maryland. find
  72. Shomir Wilson. 2013. Toward automatic processing of English metalanguage. In Proc. of IJCNLP.
  73. Changbing Yang, Franklin Ma, Freda Shi, and Jian Zhu. 2025. LingGym: How far are LLMs from thinking like field linguists? In Proc. of EMNLP, pages 1314– 1340, Suzhou, China. find
  74. Olga Zamaraeva. 2016. Inferring morphotactics from interlinear glossed text: Combining clustering and precision grammars. In Proc. of the SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 141–150, Berlin, Germany. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schneider-2026-speaking-language-reflections-metalanguage-research/