📄 Comparing Human and Machine Performance for Natural Language Information Extraction: Results for English Microelectronics From the MUC-5 Evaluation

Summary

The paper asks how close automatic information extraction systems were, in 1993, to human analysts doing the same job. The setting is the English Microelectronics task of MUC-5/Tipster, where facts from news and trade-magazine articles are filled into object-oriented templates. The author describes how the reference templates ("keys") were built and what quality controls were used: detailed fill rules, a two-analyst coding procedure, and checking software. He also studies how much the analysts' differing roles influenced the keys, compares ways of scoring humans, and tracks how analyst skill grew over time. Four experienced analysts each coded 120 test articles. Their scores were set against the three top machine systems on those same articles. Humans erred about 33% of the time and the best machine about 62%. The human error rate is itself strikingly high, which says the task is hard and loosely defined. The study matters because it gives evaluation numbers an interpretable ceiling, so scores can be read as a measure of readiness for real use.

Key points

• The template design was object-based. Slots held either set fills (categorical choices) or string fills (text copied verbatim), plus pointers to other objects. Alternative fills and unscored comments were allowed.
• Fill rules for English Microelectronics ran to a 40-page single-spaced document. A non-analyst maintained it, and a microelectronics fabrication expert helped settle technical cases.
• Two analysts coded each article independently, and one of them then merged both codings into the key. This was used for 300 high-quality development articles and all 300 test articles. The other 700 development articles had a single coder.
• Supporting tools were:
  • Locke, a template-filling GUI from New Mexico State University that offers mouse-based text copying and checks for missing or unreferenced objects;
  • the Tipster scoring program from SAIC, which listed differences between the two codings;
  • a separate syntax checker for finished keys.
• On the 120 test articles coded by all four analysts, mean human error was roughly half the machine error (33% vs. about 62% for the best system). Analysts who joined later raised their consistency with early starters by about 4.4 points between the 18- and 24-month tests.

Interesting ideas

• I1. The paper separates "quality" (real mistakes such as misreading, forgotten rules, typos, broken links) from "consistency" (legitimate disagreement over ambiguity, inference, or rule reading), so each can be targeted with a different fix.
• I2. Human performance is measured on the very same items and with the very same metric as the systems, which gives a realistic ceiling instead of an assumed 100%.
• I3. The tooling prevents errors at the source: text is copied by selection rather than retyped, and structural checks flag dangling or missing objects before the key is finished.
• I4. Keys are built by merging two blind independent codings, and recurring disagreements are fed back into the written rules.
• I5. The recall–precision view shows that a system can match humans on one axis if it gives up the other, which suggests mixed human-machine pipelines.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when witness parsers disagree on silver data, tag each disagreement as either an error or a genuine ambiguity, and only treat errors as repair-rule candidates.
• I2 (answer): have a few people answer a slice of the fable-QA or school-test sets, so Mova's score can be reported against measured human agreement.
• I3 (world): run automatic integrity checks on the world of the text, such as entities nobody refers to, missing participants, or broken event links, before answering from it.
• I4 (tree): when converting treebanks between UD dialects, reconcile two independent conversions and turn repeated conflicts into explicit conversion rules.
• I5 (frames): let frames carry confidence so that only high-precision role assignments are trusted automatically and uncertain ones go to an LLM teacher or a human.