Elamur Research⌕
☰
home / papers / 2000 / Comparing Linguistic Interpretation Schemes for English Corpora

Comparing Linguistic Interpretation Schemes for English Corpora

Paper · 2000 · Eric Atwell, George Demetriou, John Hughes et al.

Summary

The AMALGAM project (Automatic Mapping Among Lexico-Grammatical Annotation Models) addresses a practical problem in English corpus linguistics. Many annotated corpora exist, but each uses its own part-of-speech tagset and parsing scheme, which makes them hard to compare or reuse together.

The team built several resources:

  • documentation for eight PoS tagsets, illustrated with examples from their source corpora;
  • a 180-sentence multi-tagged corpus drawn from three genres and checked by experts in each scheme;
  • one shared "corpus-neutral" tokenizer;
  • a family of Brill taggers, retrained once for each scheme;
  • a MultiTreebank that shows a single sentence analysed under many parsing schemes;
  • a free email tagging service.

On their own training corpora the taggers scored well. On the mixed-genre test sample every tagger lost accuracy, which suggests the schemes are not truly generic.

The authors draw two conclusions. Tagsets are too varied to merge into one standard, and full parsing schemes are even further apart. They also argue that scoring parsers by bracket matching penalises richer analyses. The work matters because it shows how to compare annotation schemes on shared text rather than in the abstract.

Key points

  • Eight PoS tagsets compared: Brown, ICE, LLC, LOB, parts, POW, SEC and UPenn.
  • Multi-tagged corpus: 60 sentences each from COLT (teenager speech, 407 words), SEC (radio broadcasts, 2016 words) and IPSM (software manuals, 1016 words), for 3439 words in total.
  • Accuracy on source corpora: the retrained Brill tagger reached between 90.59% (ICE) and 97.43% (Brown) on 10,000 words of each source corpus.
  • Accuracy on the proofread multi-tagged corpus: results were lower across the board (Brown 94.3, UPenn 93.1, ICE 89.6, parts 86.7, LLC 86.6, POW 86.4). The COLT speech data was generally the hardest.
  • MultiTreebank: the same sentence is parsed under ENGCG, L-ATR, LOB-TREEBANK, ALICE, POW, SEC, Despar, SUSANNE and Principar.
  • Scoring parsing schemes: schemes were rated against the EAGLES syntactic annotation layers (a–h).
  • Email service usage: running since December 1996, it handled 19,839 messages and over 628 MB of text by December 1999.

Interesting ideas

  • I1. Instead of writing rules that translate one tagset into another, the authors retagged the raw words with a tagger trained on the target scheme. Earlier attempts had shown that one-to-many and many-to-many tag correspondences caused more errors than simply ignoring the source tags.
  • I2. A single shared tokenizer was used for all schemes to keep comparisons fair, and the known clashes it causes were listed explicitly (enclitics, possessives, multiword expressions such as "for example").
  • I3. Taggers were tested not only on held-out data from their own corpus but also on an expert-proofread sample spanning very different genres. The accuracy drop on that sample served as a measure of how generic each scheme really is.
  • I4. ICE tags write the major category first, with features in brackets after it, such as N(com,sing). Users found this format more intuitive, and it can be reduced to coarse categories just by dropping the brackets.
  • I5. The EAGLES layers act as a checklist for profiling what a scheme encodes: bracketing, labels, dependencies, functions, subclasses, deep information, rank, and spoken-language features. The checklist makes it clear that comparing schemes only on bracketing is unfair.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): when converting between UD dialects or legacy treebanks, where a relation maps ambiguously, reparse from tokens with a target-dialect model rather than forcing hand-written mapping rules.
  • I2 (tokens): keep one canonical tokenization for crate en and maintain a documented table of where each external corpus tokenizes differently, so mismatches are not counted as tagger errors.
  • I3 (learn): add a small hand-checked cross-genre set (speech-like text, manuals, tales) to evaluation, and treat the gap from in-domain scores as an explicit metric.
  • I4 (tokens): keep UPOS and FEATS cleanly separable in every explanation shown to users, so a coarse view and a fine view come from the same annotation.
  • I5 (learn): profile each witness parser by which layers it actually provides, and weight its votes in silver-data agreement only on the layers it really encodes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of LeedsLeeds · United Kingdom42 works · 1,823 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,379 works

Cite

Cite as Eric Atwell, George Demetriou, John Hughes, Amanda Schiffrin, Clive Souter and Sean Wilcock. 2000. Comparing Linguistic Interpretation Schemes for English Corpora. Proceedings of the COLING-2000 Workshop on Linguistically Interpreted Corpora.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{atwell2000comparing,
  title = {{Comparing Linguistic Interpretation Schemes for English Corpora}},
  author = {Eric Atwell, George Demetriou, John Hughes, Amanda Schiffrin, Clive Souter and Sean Wilcock},
  booktitle = {Proceedings of the COLING-2000 Workshop on Linguistically Interpreted Corpora},
  year = {2000},
  doi = {10.13140/2.1.4391.5203},
  url = {https://doi.org/10.13140/2.1.4391.5203}
}

References

46 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aarts, Jan. 1996. A tribute to W. Nelson Francis and Henry Kucera: grammatical annotation. ICAME Journal 20:104-107. find
  2. Aarts, Jan, Hans van Halteren and Nelleke Oostdijk. 1996. The TOSCA analysis system. In C. Koster and E Oltmans (eds). Proceedings of the first AGFL workshop. 181-191. Technical Report CSI-R9604, Computing Science Institute, University of Nijmegen. find
  3. Andersen, Gisle, and Anna-Brita Stenstrom. 1996. COLT: a progress report. ICAME Journal 20:133-136. find
  4. Atwell, Eric. 1982. LOB Corpus tagging project: post-edit handbook. Department of Linguistics and Modern English Language, University of Lancaster. find
  5. Atwell, Eric. 1983. Constituent Likelihood grammar. ICAME Journal 7:34-66. find
  6. Atwell, Eric. 1996. Comparative evaluation of grammatical annotation models. In (Sutcliffe et al 1997), 25-46. find
  7. Atwell, Eric, John Hughes and Clive Souter. 1994. AMALGAM: Automatic Mapping Among Lexico-Grammatical Annotation Models. In Judith Klavans and Philip Resnik (eds.), The balancing act - combining symbolic and statistical approaches to language. Proceedings of the workshop in conjunction with the 32nd annual meeting of the Association for Computational Linguistics. New Mexico State University, Las Cruces, New Mexico, USA. find
  8. Atwell, Eric, George Demetriou, John Hughes, Amanda Schiffrin, Clive Souter, and Sean Wilcock. 2000. A comparative evaluation of modern English corpus grammatical annotation schemes. ICAME Journal 24, to appear. find
  9. Belmore, Nancy. 1991. Tagging Brown with the LOB tagging suite. ICAME Journal 15:63-86. find
  10. Benello, J., A. Mackie and J. Anderson. 1989. Syntactic category disambiguation with neural networks. Computer Speech and Language 3:203-217. find
  11. Black, William, and Philip Neal. 1996. Using ALICE to analyse a software manual corpus. In (Sutcliffe et al 1996), 47-56. find
  12. Booth, Barbara. 1985. Revising CLAWS. ICAME Journal 9:29-35. find
  13. Brill, Eric. 1993. A Corpus-based approach to language learning. PhD thesis, Department of Computer and Information Science, University of Pennsylvania. find
  14. Briscoe, Edward and John Carroll. 1993. Generalised probabilistic LR parsing of natural language (corpora) with unification-based grammars. Computational Linguistics 19:25-60. find
  15. EAGLES (1996), WWW site for European Advisory Group on Language Engineering Standards, http://www.ilc.pi.cnr.it/EAGLES96/home.html Specifically: Leech, Geoffrey, Ruthanna Barnett and Peter Kahrel, EAGLES Final Report and guidelines for the syntactic annotation of corpora, EAGLES Report EAG-TCWG-SASG/1.5. ↗
  16. Eeg-Olofsson, Mats. 1991. Word-class tagging: Some computational tools. PhD thesis. Department of Linguistics and Phonetics, University of Lund, Sweden. find
  17. Elliott, John, and Eric Atwell. 2000. Is there anybody out there?: the detection of intelligent and generic language-like features. In Journal of the British Interplanetary Society, 53:1/2, 13-22. find
  18. Elliott, John, Eric Atwell, and Bill Whyte. 2000. Language identification in unknown signals. Proc COLING’2000. find
  19. Garside, Roger. 1996. The robust tagging of unrestricted text: the BNC experience. In Jenny Thomas and Mick Short (eds) Using corpora for language research: studies in the honour of Geoffrey Leech, 167-180. London: Longman. find
  20. Greene, Barbara and Gerald Rubin. 1981. Automatic grammatical tagging of English. Providence, R.I.: Department of Linguistics, Brown University. find
  21. Greenbaum, Sidney. 1993. The tagset for the International Corpus of English. In Clive Souter and Eric Atwell (eds) Corpus-based Computational Linguistics. 11-24. Amsterdam: Rodopi. find
  22. Grefenstette, Gregory. 1996. Using the SEXTANT low-level parser to analyse a software manual corpus. In (Sutcliffe et al 1996), 139-158. find
  23. Hughes, John and Eric Atwell. 1994. The automated evaluation of inferred word classifications. In Anthony Cohn (ed.), Proceedings of the European Conference on Artificial Intelligence (ECAI). 535-539. Chichester, John Wiley. find
  24. Hughes, John, Clive Souter and Eric Atwell. 1995. Automatic extraction of tagset mappings from parallel-annotated corpora. In From texts to tags: issues in multilingual language analysis. Proceedings of SIGDAT workshop in conjunction with the 7th Conference of the European Chapter of the Association for Computational Linguistics. University College Dublin, Ireland. find
  25. Johansson, Stig, Eric Atwell, Roger Garside and Geoffrey Leech. 1986. The Tagged LOB corpus: users’ manual. Bergen University, Norway: ICAME, The Norwegian Computing Centre for the Humanities. Available from http://www.hit.uib.no/icame/lobman/lob-cont.html ↗
  26. Karlsson, Fred, Atro Voutilainen, Juha Heikkila, and Arto Anttila. 1995. Constraint Grammar: a language-independent system for parsing unrestricted text. Berlin: Mouton de Gruyter. find
  27. Kyto, Merja and Atro Voutilainen. 1995. Applying the Constraint Grammar parser of English to the Helsinki corpus. ICAME Journal 19:23-48. find
  28. Leech, Geoffrey, Roger Garside and Eric Atwell. 1983. The automatic grammatical tagging of the LOB corpus. ICAME Journal 7:13-33. find
  29. Lin, Dekang. 1994. PRNCIPAR – an efficient, broad-coverage, principle-based parser. Proceedings of COLING-94, Kyoto. 482-488. find
  30. Lin, Dekang. 1996. Using PRINCIPAR to analyse a software manual corpus. In (Sutcliffe et al 1996), 103-118. man 1986. parts. The on-line Unix manual. find
  31. Marcus, Mitch, M Marcinkiewicz, and Barbara Santorini. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics 19:313-330.
  32. O'Donoghue, Tim. 1991. Taking a parsed corpus to the cleaners: the EPOW corpus. ICAME Journal 15:55-62 find
  33. Oostdijk, Nelleke. 1996. Using the TOSCA analysis system to analyse a software manual corpus. In (Sutcliffe et al 1996), 179-206. find
  34. Osborne, Miles. 1996. Using the Robust Alvey Natural Language Toolkit to analyse a software manual corpus. In (Sutcliffe et al 1996), 119-138. find
  35. Owen, M. 1987. Evaluating automatic grammatical tagging of text. ICAME Journal 11:18-26. find
  36. Qiao, Hong Liang and Renje Huang. 1998. Design and implementation of AGTS probabilistic tagger. ICAME Journal 22: 23-48. find
  37. Santorini, Barbara. 1990. Part-of-speech tagging guidelines for the Penn Treebank project. Technical report MS-CIS-90-47. University of Pennsylvania: Department of Computer and Information Science. find
  38. Sleator, D. and Temperley, D. 1991. Parsing English with a Link grammar. Technical Report CMU-CS-91-196. School of Computer Science, Carnegie Mellon University.
  39. Souter, Clive. 1989a. The COMMUNAL project: extracting a grammar from the Polytechnic of Wales corpus. ICAME Journal 13:20-27. find
  40. Souter, Clive. 1989b A short handbook to the Polytechnic of Wales Corpus. Bergen University, Norway: ICAME, The Norwegian Computing Centre for the Humanities. Available from http://kht.hit.uib.no/icame/manuals/pow.html ↗
  41. Sutcliffe, Richard, Heinz-Detlev Koch and Annette McElligott (eds.). 1996. Industrial parsing of software manuals. Amsterdam: Rodopi. find
  42. Sutcliffe, Richard, and Annette McElligott. 1996. Using the Link parser of Sleator and Temperley to analyse a software manual corpus. In (Sutcliffe et al 1996), 89-102. find
  43. Taylor, Lolita and Gerry Knowles. 1988. Manual of information to accompany the SEC corpus: The machine readable corpus of spoken English. University of Lancaster: Unit for Computer Research on the English Language. Available from http://kht.hit.uib.no/icame/manuals/sec/INDEX.HTM ↗
  44. Teahan, Bill. 1998. Modelling English text. PhD Thesis, Department of Computer Science, University of Waikato, New Zealand. find
  45. Ting, Christopher, and Peh Li Shiuan. 1996. Using a dependency structure parser without any grammar formalism to analyse a software manual corpus. In (Sutcliffe et al 1996), 159-178. find
  46. Voutilainen, Atro, and Timo Jarvinen. 1996. Using the English Constraint Grammar Parser to analyse a software manual corpus. In (Sutcliffe et al 1996), 57-88. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/atwell-2000-comparing-linguistic-interpretation-schemes-english/