Elamur Research⌕
☰
home / papers / 2013 / Toward Automatic Processing of English Metalanguage

Toward Automatic Processing of English Metalanguage

Paper · 2013 · Shomir Wilson

Summary

Metalanguage is language about language: sentences that talk about a word, a name, a spelling or a quoted phrase. Such sentences carry direct information about language, yet they confuse parsers and fit poorly into word-sense frameworks. Wilson reports the first attempt to find one common type automatically in English, called mentioned language. The data come from an earlier corpus built from English Wikipedia: 629 mention sentences and 1,764 non-mention sentences.

The work has two parts:

  • Detection. The task is to tell whether a sentence contains a mention. Bag-of-words baselines in Weka were compared with a feature set reduced to about ten high-information "metawords". Its performance roughly reaches the ceiling implied by inter-annotator agreement.
  • Delineation. The task is to mark the exact mentioned span. This was explored through case studies of three metawords: tree patterns over constituency parses for term and word, and semantic role labels for call.

The span patterns were precise but reliable only as far as the underlying parser or SRL. The results show that automatic processing of metalanguage is feasible.

Key points

  • Corpus. It was built by filtering Wikipedia sentences that have a stylistic cue (bold, italics, quotes) near one of 8,735 "mention-significant" words and collocations taken from WordNet. A human then labeled the candidates.

  • Reliability. Three extra annotators labeled 100 sentences, and Kappa between each pair of them averaged 0.74. Kappa between the primary annotator and their majority vote was 0.90.

  • Detection setup. Four feature sets (SW, UW, SWSB, UWUB) were run with five classifiers (Naive Bayes, SMO, J48, IBk, Decision Table) under ten-fold cross validation. The MW set kept the top ten stemmed words by information gain, chosen separately inside each training fold.

  • Delineation results.

    MetawordMethodSentencesExact spansErrors
    term (noun)TRegex apposition patterns on Stanford Parser output9157 of 59 (~97%)2 underlabeled
    word (noun)TRegex apposition patterns on Stanford Parser output10757 of 64 (~89%)3 overlabeled, 4 underlabeled
    call (verb)Illinois SRL "attribute of arg1"15868 of 85 (~80%)16 overlabeled, 1 underlabeled

  • Detection via patterns. Pattern firing had precision 1.0 for term and word. Most errors traced back to parser or SRL mistakes. The three case studies together covered only 33% of the corpus's mention sentences.

Interesting ideas

  • I1. A tiny lexicon of about ten information-gain-selected "metawords" can stand in for a full vocabulary when flagging that a sentence talks about language itself; it is cheap and readable.
  • I2. A single structural pattern does two jobs at once: when it matches, it marks the exact span and also counts as evidence that the sentence contains a mention.
  • I3. Spans can be scored as exact, overlabeled or underlabeled, kept apart from whether the pattern fired at all, so that boundary errors and detection errors are not mixed together.
  • I4. Semantic role arguments tend to "spill" past the true mention boundary, as when a trailing for-clause gets absorbed into the naming role.
  • I5. Mentioned language is direct, extractable knowledge about language, for example "graupel refers to a kind of precipitation" or "the name is actually Rolla".

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Tag tokens near metawords such as term, word, called or spelled with a "possibly mentioned" feature, so that later stages do not force ordinary senses or morphology on them.
  • I2 (tree): Write explainable UD patterns, such as term plus an appositive, or call plus an xcomp or object predicate, that both detect and delimit mentioned spans.
  • I3 (learn): Use this three-way scope breakdown when evaluating span-producing stages such as SRL arguments, quotes and named spans, to see whether failures come from boundaries or from recognition.
  • I4 (frames): When filling a naming frame from VerbNet roles, trim the predicate argument to its core noun phrase and drop adjunct clauses, so that X is called Y yields a clean name Y.
  • I5 (world): Store detected definitions, alternate names and spellings as lexical facts or alias links for entities in the text, rather than treating the mentioned word as a real-world referent.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Shomir Wilson. 2013. Toward Automatic Processing of English Metalanguage. Proceedings of the Sixth International Joint Conference on Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{wilson2013toward,
  title = {{Toward Automatic Processing of English Metalanguage}},
  author = {Shomir Wilson},
  booktitle = {Proceedings of the Sixth International Joint Conference on Natural Language Processing},
  year = {2013},
  url = {https://aclanthology.org/I13-1091/}
}

References

28 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adler, B. T., de Alfaro, L., Pye, I., & Raman, V. (2008). Measuring author contributions to the Wikipedia. In Proceedings of the 4th International Symposium on Wikis (pp. 15:1–15:10). New York, NY, USA: ACM. doi:10.1145/1822258.1822279 ↗
  2. Aha, D. W., & Kibler, D. (1991). Instance-based learning algorithms. In Machine Learning (pp. 37– 66). ↗
  3. Anderson, M. L., Fister, A., Lee, B., & Wang, D. (2004). On the frequency and types of meta-language in conversation: A preliminary report. In 14th Annual Conference of the Society for Text and Discourse. find
  4. Anderson, M. L., Okamoto, Y. A., Josyula, D., & Perlis, D. (2002). The Use-Mention Distinction and Its Importance to HCI. In Proceedings of the Sixth Workshop on the Semantics and Pragmatics of Dialog, 21–28. find
  5. Audi, R. (1995). The Cambridge Dictionary of Philosophy. Cambridge University Press. find
  6. Bird, S. (2006). NLTK: The natural language toolkit. In Proceedings of the COLING/ACL on Interactive Presentation Sessions (pp. 69–72). Stroudsburg, PA, USA: Association for Computational Linguistics.
  7. Cappelen, H., & Lepore, E. (1997). Varieties of quotation. Mind, 106(423), 429–450. doi:10.1093/mind/106.423.429 ↗
  8. Davidson, D. (1979). Quotation. Theory and Decision, 11(1), 27–40. doi:10.1007/BF00126690 ↗
  9. Fellbaum, C. (1998). WordNet: An electronic lexical database. Cambridge: MIT Press. find
  10. García-Carpintero, M. (2004). The deferred ostension theory of quotation. Noûs, 38(4), 674–692. ↗
  11. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009). The WEKA data mining software: An update. ACM SIGKDD Explorations Newsletter, 11, 10–18. doi:10.1145/ 1656274.1656278 ↗
  12. Hu, G. (2010). A place for metalanguage in the L2 classroom. ELT Journal. doi:10.1093/elt/ccq037 ↗
  13. Jaworski, A., Coupland, N., & Galasinski, D. (Eds.). (2004). Metalanguage: Language, Power, and Social Process. De Gruyter. find
  14. John, G., & Langley, P. (1995). Estimating Continuous Distributions in Bayesian Classifiers. In Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence (pp. 338–345). Morgan Kaufmann. find
  15. Keerthi, S. S., Shevade, S. K., Bhattacharyya, C., & Murthy, K. R. K. (2001). Improvements to Platt’s SMO Algorithm for SVM Classifier Design. Neural Computation, 13, 637–649. doi:10.1162/ 089976601300014493 ↗
  16. Kohavi, R. (1995). The Power of Decision Tables. In Proceedings of the European Conference on Machine Learning (pp. 174–189). Springer Verlag. find
  17. Levy, R., & Andrew, G. (2006). TRegex and TSurgeon: tools for querying and manipulating tree data structures. In Proceedings of the Fifth International Conference on Language Resources and Evaluation. find
  18. Maier, E. (2007). Mixed quotation: between use and mention. In Logic and Engineering of Natural Language Semantics Workshop. Retrieved from http://ncs.ruhosting.nl/emar/em_lenls_quot.pdf ↗
  19. Marneffe, M. D., MacCartney, B., & Manning, C. D. (2006). Generating typed dependency parses from phrase structure parses. In Proceedings of the International Conference on Language Resources and Evaluation, pp. 449–454.
  20. Punyakanok, V., Roth, D., & Yih, W. (2008). The importance of syntactic parsing and inference in semantic role labeling. Computational Linguistics, 34(2), 257–287. doi:10.1162/coli.2008.34.2.257 ↗
  21. Quine, W. V. O. (1940). Mathematical logic. Cambridge, MA: Harvard University Press. find
  22. Quinlan, J. (1993). C4.5: Programs for Machine Learning. San Mateo, Calif.: Morgan Kaufmann Publishers.
  23. Saka, P. (2006). The Demonstrative and Identity Theories of Quotation. Journal of Philosophy, 103(9), 452–471. ↗
  24. Sperber, D., & Wilson, D. (1981). Irony and the Use-Mention Distinction. In Radical Pragmatics (pp. 295– 318). New York. find
  25. Tarski, A. (1933). The concept of truth in formalized languages. In J. H. Woodger (Ed.), Logic, Semantics, Mathematics. Oxford: Oxford University Press. find
  26. Wilson, S. (2011a). A computational theory of the use-mention distinction in natural language. University of Maryland at College Park. PhD Thesis, College Park, MD, USA. find
  27. Wilson, S. (2011b). In search of the use-mention distinction and its impact on language processing tasks. International Journal of Computational Linguistics and Applications, 2(1-2), 139–154. find
  28. Wilson, S. (2012). The Creation of a Corpus of English Metalanguage. In Proceedings of the 50 th Annual Meeting of the Association for Computational Linguistics, (pp. 638–646).

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wilson-2013-automatic-processing-english-metalanguage/