📄 Named Entity Recognition without Gazetteers

Summary

The paper asks how much a Named Entity recognition system really depends on large gazetteers, which are often described as a costly bottleneck. The authors use the MUC-7 data: New York Times Service articles, 200 for training and 100 for testing. First they measure how far pure list lookup can go. They try lists harvested from training texts, lists of widely known entities taken from public sources, and both combined. Then they evaluate their own MUC-7 system, one of the top scorers in that competition. It mixes rule-based grammars with maximum entropy models and uses both the internal shape of a name and the context around it. They run this system with gazetteers of different types and sizes. Without any lists, persons and organisations still score in the high eighties. Locations suffer, but a small list of country names lifts them to the mid-eighties, and lists built cheaply from 30 in-domain articles make the system competitive again. The main lesson is that a few well-known names matter more than huge lists of rare ones. The authors note that these findings cover only mixed-case journalistic English.

Key points

• Lookup baseline with lists from the 200 MUC-7 training articles (1228 person, 809 organisation and 770 location names) gave about 90% precision and 40–70% recall. Recall/precision were: organisation 49/75, person 26/92, location 76/93.
• Public lists (5000 locations from the CIA World Fact Book, 33,000 organisations from financial websites, 27,000 famous people) worked about as well for locations. They were slightly worse for persons and very poor for organisations (recall 3, precision 51). Combining both list types raised recall but lowered precision.
• The full MUC-7 system combines "sure-fire" contextual rules with maximum entropy models. It relies on internal (phrasal) and external (contextual) evidence, following McDonald (1996). One example rule: a capitalised sequence followed by "a former director" marks a PERSON.
• With no gazetteers, organisation and person recognition stays in the high eighties. Without lists, well-known names that appear with little context are missed, for example "Bonn" in "the Bonn government" or NYSE.
• For comparison, Isoquest used 110,000 names. Krupka and Hausman (1998) found little loss when cutting this to 25,000 or 9,000 names, but a big gain from adding just 42 entries.

Interesting ideas

• I1. Concentrate on famous names. A gazetteer is most useful for entities that writers assume readers already know, because those appear without explanatory context. Lists can therefore stay small and target high-frequency names. This is a cheap strategy for low-resource languages, including Ukrainian.
• I2. Grow gazetteers from in-domain text. Collect entities from a small sample of documents in the target domain, with minimal cleanup, and use them as lists. Thirty articles were already enough to recover performance, which suggests a cheap bootstrap for new genres.
• I3. Treat list entries as soft hints. First names were stored only as "likely" first names rather than definite ones, since a word like "Bill" can mean other things. Letting the model weigh list entries as evidence, instead of applying them as hard labels, reduces errors from ambiguous names.
• I4. Apply high-precision context patterns first. Very reliable patterns, such as a profession, an age in commas, or "shares in" + Name, can be applied before statistical decisions. This gives trustworthy anchors that later stages can build on.
• I5. Pick metrics that fit the use case. For document retrieval, finding one mention of each entity per document may be enough, so low recall can be acceptable. Choosing metrics by the downstream task helps avoid over-engineering.