📄 FreeLing: An Open-Source Suite of Language Analyzers

Summary

Most NLP applications, such as machine translation, summarization and dialogue systems, depend on basic analysis steps like tokenization, morphology, lemmatization and PoS tagging. Developers spend a large share of their effort adapting existing tools to their own platform, language, format or API. The authors present FreeLing, a C++ library of such analyzers released as free software under the LGPL. It builds on their earlier client-server object architecture (Carreras and Padró, 2002) and fixes that design's rigid, predefined server APIs. Applications call the analyzers directly as library functions or reach them remotely through CORBA. They exchange data through a small set of linguistic classes (analysis, word, sentence) and do not need text interchange formats.

The first release covers English, Spanish and Catalan. It handles tokenization, sentence splitting, morphological analysis, multiword, number, date, currency and proper-noun detection, and trigram HMM PoS tagging. Tagging precision is above 95%. The authors position the suite as a reusable, extensible base for both research and commercial systems.

Key points

• The architecture has two kinds of objects:
  • Linguistic data classes: analysis holds a lemma, PoS tag and probability; word holds a form plus its possible analyses, including multiword support; sentence is a list of words.
  • Processing classes: tokenizer, splitter, morfo and tagger.
• morfo runs a cascade of sub-modules: locutions, dictionary with suffix handling, numbers, dates, quantities, punts, probabilities and np.
• The dictionaries differ by language:
  • English: over 160,000 forms for about 78,000 lemmas derived from WSJ, with the 200 most frequent forms hand-checked.
  • Spanish: about 71,000 hand-coded forms.
  • Catalan: about 46,000 hand-coded forms.
  • Both Spanish and Catalan include all closed-class lemmas plus the 5,000 most frequent open-class lemmas.
• The PoS tagger is a trigram HMM in the Cutting et al. / Brants (TnT) style. It was trained on WSJ for English and on a 100,000-word hand-disambiguated corpus for Spanish and Catalan, and reaches over 95% precision for all three languages.
• Speed on a P4 2.8 GHz machine:
  • morphological analysis alone: about 6,000 words/s;
  • tagging alone: 3,100 words/s;
  • both together: 2,300 words/s.

Interesting ideas

• I1. Tunable ambiguity for unknown words: Out-of-vocabulary words get candidate tags from suffix-conditioned probabilities, and a probability threshold controls how many tags are kept. With no threshold, a word receives every open tag, up to 91 per word with the rich Spanish/Catalan tagset, which makes the tagger slow and less accurate. A moderate threshold trades a little recall for much better speed and precision, a pattern useful for any morphologically rich language.
• I2. Client owns its data format: The library imposes only a few in-memory classes. Each application keeps its own storage format and converts only at the library boundary. This avoids repeatedly serializing and parsing text between pipeline stages.
• I3. Pick-and-choose pipeline: Every module is a separate processing class. A client can run just a date recognizer, or call the morphological analyzer on already-split text. This makes it easy to reuse single components without adopting the whole stack.
• I4. Analyzer as a cascade of small recognizers: Morphology is split into independent detectors for numbers, dates, quantities, multiwords and proper nouns, followed by dictionary lookup. This makes it simple to add or replace one detector for a new language or domain.
• I5. Configuration over code for new languages: The authors plan to define language-specific modules, such as date/time patterns, in rule files rather than hard-coded logic. This approach lowers the cost of porting a toolkit to a new language.