Elamur Research⌕
☰
home / papers / 2012 / The Creation of a Corpus of English Metalanguage

The Creation of a Corpus of English Metalanguage

Paper · 2012 · Shomir Wilson

Summary

The paper addresses metalanguage, meaning language used to talk about language itself, as in sentences that discuss a word rather than its referent. NLP tools largely ignore this. Parsers mis-tag mentioned words, sentiment systems misread quoted terms, and dialog systems fail when a user corrects a misrecognized word. Wilson focuses on the tractable subtype called mentioned language. He gives a formal definition: tokens whose purpose is to highlight some feature of the tokens themselves or of their type. Because annotators found that definition too abstract, he adds a practical substitution rubric. Using English Wikipedia, whose writers often mark mentioned words with quote marks, italics or bold, he builds three successively richer corpora. The last, the Enhanced Cues Corpus, is the first to mark the exact word span that is mentioned. Expert re-annotation shows good agreement on whether mention is present but weaker agreement on its subtype. The resource lays groundwork for automatically detecting and delineating metalanguage.

Key points

  • Mentioned language is defined as a token or token set produced to highlight a property of that token or its type, such as spelling, pronunciation, meaning, connotation or source.
  • The annotation rubric replaces a candidate X with "that [item]" (word, name, phrase, letter…). X counts as mentioned if the sentence's truth conditions stay the same when the phrase points back to X.
  • The source is English Wikipedia. Stylistic cues (quotes, italic, bold) were used to collect candidates, which were then labeled with five categories: WW, NN, SP, OM and XX (not mentioned).
  • Reliability check: three extra expert annotators labeled 100 sampled instances. All four annotators agreed on 46 positives and 30 negatives, with an average pairwise Kappa of 0.74 for presence of mention.
  • Context and outlook:
    • The Stanford Parser tags go in "The button labeled go" as VB.
    • An earlier BNC-based study (Anderson et al. 2004) found metalanguage in about 11% of spoken utterances, but it lacked span-level labels.
    • Preliminary detection with bag-of-words plus a decision tree catches about 70% of instances.

Interesting ideas

  • I1. A deictic substitution test ("replace X with that word and check whether the meaning holds") turns a vague semantic notion into a repeatable judgment that annotators can apply.
  • I2. Mentioned spans often attach to a small, closed set of mention-significant nouns (word, name, term) through apposition, or fill specific roles of verbs like call and spell.
  • I3. Stylistic cues (quotes, italics, bold) act as a cheap high-recall filter for mining candidates, which humans then judge. Redacting those cues during annotation probes how much judgments depend on them.
  • I4. Agreement can be measured twice: once on a coarse binary question (is this mention at all?) and once on fine subtypes, which shows where annotator consensus breaks down.
  • I5. Words that appear inside mention or quotation do not express the writer's own stance. A user turn like "No, I said 'Arlington'" is a correction about a word, not new content.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Run the "that [item]" check when filling frame roles, so a span like Mississippi in "fun to spell" becomes a linguistic-token argument rather than a geographic entity.
  • I2 (tree): Add repair rules that re-attach and re-tag the mentioned span as a single quoted unit after nouns like verb or word, fixing parses such as go/VB in "labeled go".
  • I3 (learn): Mine Wikipedia-style highlighted spans as silver candidates for mention detection, and use cue-stripped copies as a negative-control test of whether the model relies only on punctuation.
  • I4 (learn): Score witness-parser or LLM-teacher agreement on collapsed labels and fine labels separately, to see whether disagreement sits in detection or in classification.
  • I5 (world): Store mentioned strings as language objects linked to their referents, so sentiment, attribution and dialog-repair reasoning do not treat a quoted term as a fact or opinion of the speaker.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Shomir Wilson. 2012. The Creation of a Corpus of English Metalanguage. Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{wilson2012creation,
  title = {{The Creation of a Corpus of English Metalanguage}},
  author = {Shomir Wilson},
  booktitle = {Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2012},
  url = {https://aclanthology.org/P12-1067/}
}

References

24 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adler, B. Thomas, Luca de Alfaro, Ian Pye & Vishwanath Raman. 2008. Measuring author contributions to the Wikipedia. In Proc. of WikiSym '08. New York, NY, USA: ACM. find
  2. American Psychological Association. 2001. Publication Manual of the American Psychological Association. 5th ed. Washington, DC: American Psychological Association. find
  3. Anderson, Michael L, Yoshi A Okamoto, Darsana Josyula & Donald Perlis. 2002. The use-mention distinction and its importance to HCI. In Proc. of EDILOG 2002. 21–28. find
  4. Anderson, Michael L., Andrew Fister, Bryant Lee & Danny Wang. 2004. On the frequency and types of meta-language in conversation: A preliminary report. In Proc. of the 14th Annual Conference of the Society for Text & Discourse. find
  5. Cappelen, H & E Lepore. 1997. Varieties of quotation. Mind 106(423). 429 –450. ↗
  6. Chicago Editorial Staff. 2010. The Chicago Manual of Style. 16th ed. University of Chicago Press. find
  7. Clark, Herbert H. & Edward F. Schaefer. 1989. Contributing to discourse. Cognitive Science 13(2). 259–294. ↗
  8. Davidson, Donald. 1979. Quotation. Theory and Decision 11(1). 27–40. find
  9. Fellbaum, Christiane. 1998. WordNet: An Electronic Lexical Database. Cambridge: MIT Press. find
  10. García-Carpintero, Manuel. 2004. The deferred ostension theory of quotation. Noûs 38(4). 674–692. ↗
  11. Klein, Dan & Christopher D. Manning. 2003. Fast exact inference with a factored model for natural language parsing. Advances in Neural Information Processing Systems 15. find
  12. Loper, Edward & Steven Bird. 2002. NLTK: The Natural Language Toolkit. In Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics 1. 63–70. Association for Computational Linguistics. ↗
  13. Maier, Emar. 2007. Mixed quotation: Between use and mention. In Proc. of LENLS 2007. find
  14. Partee, Barbara. 1973. The syntax and semantics of quotation. In Stephen Anderson & Paul Kiparsky (eds.), A Festschrift for Morris Halle. New York: Holt, Rinehart, Winston. find
  15. Perlis, Donald, Khemdut Purang & Carl Andersen. 1998. Conversational adequacy: Mistakes are the essence. International Journal of Human-Computer Studies 48(5). 553–575. ↗
  16. Quine, W. V. O. 1940. Mathematical Logic. Cambridge, MA: Harvard University Press. find
  17. Raux, Antoine, Brian Langner, Dan Bohus, Alan W Black & Maxine Eskenazi. 2005. Let’s Go public! Taking a spoken dialog system to the real world. In Proc. of Interspeech 2005. find
  18. Saka, Paul. 1998. Quotation and the use-mention distinction. Mind 107(425). 113 –135. ↗
  19. Saka, Paul. 2003. Quotational constructions. Belgian Journal of Linguistics 17(1). find
  20. Strunk, Jr. & E. B. White. 1979. The Elements of Style, Third Edition. Macmillan. find
  21. Tarski, Alfred. 1933. The concept of truth in formalized languages. In J. H. Woodger (ed.), Logic, Semantics, Mathematics. Oxford: Oxford University Press. find
  22. Wilson, Shomir. 2010. Distinguishing use and mention in natural language. In Proc. of the NAACL HLT 2010 Student Research Workshop, 29–33. Association for Computational Linguistics. find
  23. Wilson, Shomir. 2011a. A Computational Theory of the Use-Mention Distinction in Natural Language. Ph.D. dissertation, University of Maryland at College Park. find
  24. Wilson, Shomir. 2011b. In search of the use-mention distinction and its impact on language processing tasks. International Journal of Computational Linguistics and Applications 2(1-2). 139–154. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wilson-2012-creation-corpus-english-metalanguage/