📄 Message Understanding Conference- 6: A Brief History

Summary

The paper reviews the history of the Message Understanding Conferences (MUC), a series of evaluation campaigns organized since 1987 by NRaD/NOSC with DARPA funding. The series centers on information extraction, i.e., the automatic extraction of structured facts from texts. The authors show how the task grew steadily more complex. It began with military messages, then moved to terrorism, and in MUC-5 covered joint ventures and microchip fabrication in English and Japanese. This complexity made system development increasingly costly. In response, MUC-6 introduced a set of separate tasks: named entity, template element, a "mini-MUC" scenario template, and a SemEval component for evaluating deeper text understanding (coreference, word sense disambiguation, predicate-argument structure). The results showed that systems could be adapted to a new scenario within a few weeks while achieving performance close to the best results of earlier MUCs. At the same time, most participants converged on a similar architecture: a cascade of finite-state automata that recognize patterns. The paper is important because it defined the standard tasks and metrics (recall, precision) on which subsequent NER and coreference research was built.

Key points

• History of the series: MUC-1 (1987) had no formal evaluation yet. The template to be filled had 10 slots in MUC-2 (1989), 18 in MUC-3 (1991), and 24 in MUC-4 (1992). In MUC-5 (1993, Tipster program), the joint ventures domain already had 11 templates with 47 slots, and a nested (hierarchical) template structure was introduced.
• Best overall result in MUC-5: 57% recall and 64% precision.
• Named entity in MUC-6 covers persons, organizations, locations, temporal expressions, monetary amounts, and percentages. The annotation is done in SGML using the ENAMEX and NUMEX tags.
• Template element: the best system achieved 75% recall and 86% precision. Descriptive noun phrases proved to be the weakest point, with only 38% recall and 51% precision.
• Scenario template (domain: management changes in companies): the best result was 47% recall and 70% precision, with most systems' scores falling within a narrow range. The coreference task still needs to be defined more precisely.