Elamur Research⌕
☰
home / papers / 1993 / Evaluating Message Understanding Systems: An Analysis of the Third Message Understanding Conference (MUC-3)

Evaluating Message Understanding Systems: An Analysis of the Third Message Understanding Conference (MUC-3)

Paper · 1993 · Nancy Chinchor, Lynette Hirschman and David D. Lewis

Summary

Before the early 1990s, message-processing systems were rarely tested on unseen data with objective scoring. This paper looks back at MUC-3 (May 1991), where 15 systems pulled structured facts about terrorist incidents in nine Latin American countries out of newswire text and filled database-like templates covering incident type, perpetrators, targets, date, location and effects. Systems also had to separate relevant messages from irrelevant ones. The authors were consultants to the conference, not system builders. They trace how the evaluation grew out of earlier MUCs, sort the participants into pattern-matching, syntax-driven and semantics-driven systems, and explain recall, precision and overgeneration scoring. They also describe black-box tests aimed at specific linguistic phenomena. A key contribution is applying approximate randomization to estimate statistical significance. Systems formed distinct performance clusters, so the method can tell systems apart. However, no single processing approach came out clearly superior. The work helped set rigorous, reproducible evaluation practice for NLP.

Key points

  • Scale grew sharply over MUCK-II, which had 8 systems. MUC-3 had 15 participants, 1,300 training documents (about 400,000 words, 18,000-word vocabulary) and a blind test of 100 new messages.
  • An automated, interactive scoring program replaced hand-scoring by each site. The earlier hand-scoring had produced inconsistent results across sites.
  • Every system scored lower on recall than on precision, and recall above 60% remained out of reach.
  • Pattern-matching and IR-style methods worked only when combined with linguistic analysis. Robust techniques such as filtering, skimming and partial parsing mattered for good scores.
  • Adapting a system to the task took roughly one person-year, which the authors flag as a serious portability problem.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States211 works · 69,000 citations🎓University of ChicagoChicago · United States45 works · 50,617 citations

Also on the byline: Leidos · Science Applications International Corporation.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Nancy Chinchor, Lynette Hirschman and David D. Lewis. 1993. Evaluating Message Understanding Systems: An Analysis of the Third Message Understanding Conference (MUC-3). Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@article{chinchor1993evaluating,
  title = {{Evaluating Message Understanding Systems: An Analysis of the Third Message Understanding Conference (MUC-3)}},
  author = {Nancy Chinchor, Lynette Hirschman and David D. Lewis},
  journal = {Computational Linguistics},
  year = {1993},
  url = {https://aclanthology.org/J93-3001/}
}

References

28 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Black, E.; Abney, S.; Flickinger, D.; Gdaniec, C.; Grishman, R.; Harrison, P.; Hindle, D.; Ingria, R.; Jelinek, E; Klavans, J.; Liberman, M.; Marcus, M.; Roukos, S.; Santorini, B.; and Strzalkowski, T. (1991). "A procedure for quantitatively comparing the syntactic coverage of English grammars." In… find
  2. Pacific Grove, CA. February 1991, 306-311. find
  3. Chatfield, C. (1988). Problem Solving: A Statistician's Guide. Chapman and Hall. find
  4. Chinchor, N. (1991a). "Evaluation metrics." In Proceedings, Third Message Understanding Conference (MUC-3). Morgan Kaufmann. San Mateo, CA. find
  5. Chinchor, N. (1991b). "Linguistic phenomena test experiment." In Proceedings, Third Message Understanding Conference (MUC-3). Morgan Kaufmann. San Mateo, CA. find
  6. Chinchor, N. (1992). "The statistical significance of the MUC-4 results." In Proceedings, Fourth Message Understanding Conference (MUC-4). Morgan Kaufmann. San Mateo, CA. find
  7. Cohen, P., ed. (1991). Proceedings, Workshop on AI Methodology. Amherst, MA, June. find
  8. Efron, B., and Tibshirani, R. (1991). "Statistical data analysis in the computer age." Science, 253, 390-395. find
  9. Hirschman, L. (1991a). "Evaluation for language understanding systems." In Proceedings, Workshop on AI Methodology, edited by P. Cohen, Amherst, MA, June 1991. find
  10. Hirschman, L. (1991b). "Comparing MUCK-II and MUC-3: Assessing the difficulty of different tasks." In Proceedings, Third Message Understanding Conference (MUC-3). Morgan Kaufmann. find
  11. Hirschman, L. (1992). "An adjunct test for discourse processing in MUC-4." In Proceedings, Fourth Message Understanding Conference (MUC-4). Morgan Kaufmann. find
  12. Hirschman, L.; Bates, M.; Dahl, D.; Fisher, W.; Garofolo, J.; Hunicke-Smith, K.; Pallett, D.; Pao, C.; Price, P.; and Rudnicky, A. (1992). "Multi-site data collection for a spoken language corpus." In Proceedings, International Conference on Spoken Language Processing (ICSLP-92). find
  13. Krupka, G., and Rau, L. (1992). "GE adjunct test report: Object-oriented design and scoring for MUC-4." In Proceedings, Fourth Message Understanding Conference (MUC-4). find
  14. Lehnert, W., and Sundheim, B. (1991). "An evaluation of text analysis technologies." AI Magazine, 12(3), 81-94. find
  15. Lewis, D. D. (1991). "Data extraction as text categorization: An experiment with the MUC-3 corpus." In Proceedings, Third Message Understanding Conference (MUC-3). find
  16. Lewis, D. D., and Tong, R. M. (1992). "Text filtering in MUC-3 and MUC-4." In Proceedings, Fourth Message Understanding Conference (MUC-4). Morgan Kaufmann. find
  17. Neal, J. G., and Walter, S. M., eds. (1991). find
  18. Natural Language Processing Systems Evaluation Workshop. Berkeley, CA. June 1991. find
  19. Noreen, E. W. (1989). Computer Intensive Methods for Testing Hypotheses: An Introduction. John Wiley & Sons. find
  20. Palmer, M., and Finin, T. (1990). "Workshop on the Evaluation of Natural Language Processing Systems." Computational Linguistics, 16(3), 175-181. find
  21. Price, P.; Hirschman, L.; Shriberg, E.; and Wade, E. (1992). "Subject-based evaluation measures for interactive spoken language systems." In Proceedings, Speech and Natural Language Workshop. find
  22. Arden House, NY, February 1992. find
  23. Proceedings, Third Message Understanding Conference (MUC-3). (1991). Morgan Kaufmann. San Mateo, CA. find
  24. Proceedings, Fourth Message Understanding Conference (MUC-4). (1992). find
  25. Morgan Kaufmann. San Mateo, CA. van Rijsbergen, C. J. (1979). Information Retrieval. Butterworth. find
  26. Santorini, B. (1990). "Part-of-speech tagging guidelines for the Penn TREEBANK project (3rd revision)." MS-CIS-90-47. CIS Department, University of Pennsylvania. Philadelphia, PA.
  27. Swets, J. A., ed. (1964). Signal Detection and Recognition by Human Observers. John Wiley & Sons. find
  28. Swets, J. A. (1969). "Effectiveness of information retrieval methods." American Documentation, 72-89. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chinchor-1993-evaluating-message-understanding-systems-analysis/