Elamur Research⌕
☰
home / papers / 2002 / Parallel Distributed Grammar Engineering for Practical Applications

Parallel Distributed Grammar Engineering for Practical Applications

Paper · 2002 · Stephan Oepen, Emily M. Bender, Uli Callmeier et al.

Summary

The paper examines how to control the quality of a large constraint-based grammar when it is developed simultaneously by teams in different countries. The example is an open-source HPSG grammar of Japanese. Over eight months it was improved in parallel by DFKI (Saarbrücken) and the company YY Technologies (California), while the grammar was already running in a commercial product. The authors show that even a minor fix in one lexical class can unpredictably break neighboring constructions, the semantics, or the parse-ranking heuristics. From this they derive requirements for regression testing. The solution was the [incr tsdb()] environment for competence and performance profiling. It stores annotated test suites and runs them in batch and distributed mode through different parsers, including LKB and PET. It then collects metrics for coverage, ambiguity, time, and memory, and compares profiles of different grammar versions. Fast daily checks made it possible to port the grammar from spoken language to email and to a new domain, substantially extending coverage. The work demonstrates that, given a proper engineering methodology, broad-coverage HPSG grammars are already suitable for real-world applications.

Key points

  • On representative customer corpora, the coverage of the Japanese grammar rose from roughly 40% to over 90%. The grammar was ported from the appointment-scheduling domain (VerbMobil) to customer-support requests submitted by email.
  • Over five months, coverage on the "banking" data increased by more than 70% in relative terms. The grammar itself grew only moderately: the number of types by about 15%, the number of rules by about 20%.
  • As of June 2002, [incr tsdb()] had been integrated with nine grammar development and parsing systems, among them LKB, PET, LiLFeS, TRALE, and PAGE. The environment has three components: a DBMS, a batch control and statistics kernel with an API for C and Lisp, and a graphical interface in Tcl/Tk.
  • ChaSen was used for tokenization. The average input length was about 50 lexical items for the "banking" set and 80 for "trading". The grammar typically produced fewer than ten parses per sentence.
  • The grammar in the product was updated once or twice a week. At the start of each working day, the developers spent 10–20 minutes obtaining fresh profiles on the shared test suites.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations🔬German Research Centre for Artificial IntelligenceKaiserslautern · Germany77 works · 29,479 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Syntax & grammar formalisms · 214 works

Cite

Cite as Stephan Oepen, Emily M. Bender, Uli Callmeier, Dan Flickinger and Melanie Siegel. 2002. Parallel Distributed Grammar Engineering for Practical Applications. COLING-02: Grammar Engineering and Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{oepen2002parallel,
  title = {{Parallel Distributed Grammar Engineering for Practical Applications}},
  author = {Stephan Oepen, Emily M. Bender, Uli Callmeier, Dan Flickinger and Melanie Siegel},
  booktitle = {COLING-02: Grammar Engineering and Evaluation},
  year = {2002},
  doi = {10.3115/1118783.1118791},
  url = {https://doi.org/10.3115/1118783.1118791}
}

References

18 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Asahara, M., & Matsumoto, Y. (2000). Extended models and tools for high-performance part-of-speech tagger. In Proceedings of the 18th International Conference on Computational Linguistics (pp. 21 – 27). Saarbrücken, Germany. find
  2. Butt, M., King, T. H., Niño, M.-E., & Segond, F. (1999). A grammar writer’s cookbook. Stanford, CA: CSLI Publications. find
  3. Callmeier, U. (2002). Preprocessing and encoding techniques in PET. In S. Oepen, D. Flickinger, J. Tsujii, & H. Uszkoreit (Eds.), Collaborative language engineering. A case study in efficient grammar-based processing. Stanford, CA: CSLI Publications. (forthcoming) find
  4. Carroll, J. (1994). Relating complexity to practical performance in parsing with wide-coverage unification grammars. In Proceedings of the 32nd Meeting of the Association for Computational Linguistics (pp. 287 – 294). Las Cruces, NM. find
  5. Copestake, A. (2002). Implementing typed feature structure grammars. Stanford, CA: CSLI Publications. find
  6. Erbach, G. (1991). An environment for experimenting with parsing strategies. In J. Mylopoulos & R. Reiter (Eds.), Proceedings of IJCAI 1991 (pp. 931 – 937). San Mateo, CA: Morgan Kaufmann Publishers. find
  7. Erbach, G., & Uszkoreit, H. (1990). Grammar engineering. Problems and prospects (CLAUS Report \# 1). Saarbrücken, Germany: Computational Linguistics, Saarland University. find
  8. Flickinger, D., Copestake, A., & Sag, I. A. (2000). HPSG analysis of English. In W. Wahlster (Ed.), Verbmobil. Foundations of speech-to-speech translation (Artificial Intelligence ed., pp. 321 – 330). Berlin, Germany: Springer. find
  9. Geist, A., Bequelin, A., Dongarra, J., Manchek, W. J. R., & Sunderam, V. (Eds.). (1994). PVM — parallel virtual machine. A users’ guide and tutorial for networked parallel computing. Cambridge, MA: The MIT Press. find
  10. Makino, T., Yoshida, M., Torisawa, K., & Tsujii, J. (1998). LiLFeS — towards a practical HPSG parser. In Proceedings of the 17th International Conference on Computational Linguistics and the 36th Annual Meeting of the Association for Computational Linguistics (pp. 807 – 11). Montreal, Canada. find
  11. Müller, S., & Kasper, W. (2000). HPSG analysis of German. In W. Wahlster (Ed.), Verbmobil. Foundations of speech-to-speech translation (Artificial Intelligence ed., pp. 238 – 253). Berlin, Germany: Springer. find
  12. Oepen, S., & Callmeier, U. (2000). Measure for measure: Parser cross-fertilization. Towards increased component comparability and exchange. In Proceedings of the 6th International Workshop on Parsing Technologies (pp. 183 – 194). Trento, Italy. find
  13. Oepen, S., & Flickinger, D. P. (1998). Towards systematic grammar profiling. Test suite technology ten years after. Journal of Computer Speech and Language, 12 (4) (Special Issue on Evaluation), 411 – 436. find
  14. Oepen, S., Netter, K., & Klein, J. (1997). TSNLP — Test Suites for Natural Language Processing. In J. Nerbonne (Ed.), Linguistic Databases (pp. 13 – 36). Stanford, CA: CSLI Publications. find
  15. Penn, G. (2000). Applying constraint handling rules to HPSG. In Proceedings of the first international conference on computational logic (pp. 51 – 68). London, UK. find
  16. Siegel, M. (2000). HPSG analysis of Japanese. In W. Wahlster (Ed.), Verbmobil. Foundations of speech-to-speech translation (Artificial Intelligence ed., pp. 265 – 280). Berlin, Germany: Springer. find
  17. Siegel, M., & Bender, E. M. (2002). Efficient deep processing of japanese. In Proceedings of the 19th International Conference on Computational Linguistics. Taipei, Taiwan. find
  18. Uszkoreit, H., Backofen, R., Busemann, S., Diagne, A. K., Hinkelman, E. A., Kasper, W., Kiefer, B., Krieger, H.-U., Netter, K., Neumann, G., Oepen, S., & Spackman, S. P. (1994). DISCO — an HPSG-based NLP system and its application for appointment scheduling. In Proceedings of the 15th International… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/oepen-2002-parallel-distributed-grammar-engineering-practical/