📄 A Maximum Entropy Approach to Identifying Sentence Boundaries

Summary

Reynar and Ratnaparkhi address sentence boundary detection: deciding whether each ., ? or ! in raw text actually ends a sentence. This is harder than it looks, because periods also appear in abbreviations, decimals, e-mail addresses and ellipses. Hand-written rules and exception lists never cover every case. The authors instead frame the task as binary classification with a maximum entropy model trained by Generalized Iterative Scaling. The model uses only the candidate token (split into the part before the mark and the part after it) and the words immediately to its left and right. They built two variants:

• a higher-accuracy system with hand-made lists of honorifics and corporate designators;
• a "highly portable" system whose only resource is an abbreviation list induced automatically from the annotated training data.

Trained on Penn Treebank Wall Street Journal text, the systems reach 98.8% and 98.0% accuracy on Palmer and Hearst's WSJ test set. This matches systems that need POS lexica or far more data. The approach matters because a small boundary-annotated corpus is enough to retrain it for a new genre or Roman-alphabet language.

Key points

• Training data: 39,441 sentences (898,737 words) from the WSJ part of the Penn Treebank (release 2). Sections 03–06 were excluded because they overlap the test data. On a Sun Ultra Sparc, the model needed 18 minutes to train, and processing a single WSJ article takes roughly 1.4 seconds.
• Best system results:
  • WSJ: 98.8% accuracy over 32,173 candidate marks.
  • Brown corpus: 97.9% over 61,282 candidates, with 750 false positives and 506 false negatives.
• Highly portable system results: 98.0% on WSJ and 97.5% on Brown. The drop on Brown shows that training and test genre should match.
• Learning curve: with only 500 training sentences, accuracy is 97.6% (best) and 96.5% (portable). Two naive baselines score far lower:
  • calling every candidate a boundary: 64.0%;
  • calling only token-final marks boundaries: 78.4%.
• Comparison with earlier work:
  • Palmer & Hearst's SATZ reaches 98.5% on WSJ, but it relies on POS information.
  • Riley's decision tree scores 99.8% on Brown, but it was trained on 25 million words of Brown itself.

Interesting ideas

• I1. Inducing the abbreviation lexicon from annotations. Any whitespace-delimited token in the training data that contains a period not marking a boundary is added to an abbreviation list. This replaces a hand-built resource with an automatically derived one, which helps when adapting tools to languages such as Ukrainian that lack curated lists.
• I2. Splitting the candidate token at the punctuation mark. Features are computed separately for the part before the mark and the part after it. This handles tokens with internal punctuation (e.g. "D.C.") cleanly and could carry over to tokenization or clitic splitting.
• I3. Generating features from templates. The designer only specifies the form of the information, such as "word to the left" or "prefix is an induced abbreviation". Concrete features are then produced by scanning the corpus, which keeps the model language-agnostic and cheap to retrain.
• I4. Reporting a learning curve and naive baselines. Measuring accuracy against training set size, alongside trivial baselines, tells practitioners how much annotation a new domain needs. The authors estimate that producing enough data would take only a few hours of work.
• I5. Running both a resource-rich and a resource-free variant. Reporting both systems side by side quantifies exactly what the hand-made lists add, which is a useful ablation pattern for any pipeline component.