📄 FreeLing 3.0: Towards Wider Multilinguality

Summary

FreeLing is an open-source C++ library that gives NLP application developers ready-made language analyzers, so they do not have to build basic processing from scratch. This paper describes version 3.0 and its design. The library has two layers: linguistic service modules and the client applications that call them. It uses data classes (analysis, word, sentence, paragraph, document) and processing classes that mostly share one processor interface. The pipeline runs from tokenization and morphology through PoS tagging, NER, sense annotation with UKB, chunk and rule-based dependency parsing, and co-reference. The main change in 3.0 is full UTF8 support. This made a language identifier possible, allows locale-aware POSIX regular expressions, and opened the way to non-Latin scripts such as Russian. Other additions include user-defined regex tagging, forbidden trigrams in the HMM tagger, SAMPA phonetic encoding, and configurable sense-repository mapping. The library is already used in several commercial products. Its value lies in giving a growing multilingual community a reusable, robust, industry-ready foundation for analysis.

Key points

• Licensing and adoption:
  • FreeLing is distributed under GPL and dual-licensed for commercial embedding. The first release was in 2003.
  • Version 2.2 was downloaded over 88,000 times after its September 2010 release.
  • The 3.0 alpha source was downloaded 530 times in its first five weeks.
• Languages:
  • Coverage grew from English, Spanish and Catalan to nine languages: Galician, Italian, Welsh, Portuguese, Asturian and Russian were added.
  • There is also a diachronic Spanish variant covering the 12th–16th centuries. It was applied in a study of how the verb haber evolved.
• Accuracy figures:
  • The PoS taggers reach roughly 97–98%. There are two of them: an HMM tagger following Brants (2000) and a relaxation-labelling tagger that can combine statistics with hand-written rules.
  • The capitalization-based NE detector is close to 90% accurate. The slower recognizer based on the CoNLL-2002 winning system exceeds 94%.
• Reused components: word sense disambiguation comes directly from UKB, and co-reference follows Soon et al. (2001).
• Industrial users: Ruby Reader, Vi-Clone, TextToSign, Dixio and Aport News (Russian news clustering).
• Planned work: a spelling-correction module, thread-based processors, and FOMA to replace the language-specific C++ recognizers for dates, numbers and quantities.

Interesting ideas

• I1. Shared processor interface: nearly every module enriches a sentence either in place or as a copy, using the same method signature. Modules can therefore be chained, swapped or called individually, which suits any modular pipeline, including one for Ukrainian.
• I2. Regex escape hatch for domain tokens: a user map applies a regex and assigns a fixed lemma/PoS. For example, Twitter handles can be tagged as proper nouns without retraining NER. This is a cheap way to adapt a pipeline to new domains such as social media or trade documents.
• I3. Forbidden trigrams: the tagger model lists tag sequences that are linguistically impossible and keeps them out of smoothing, so their probability stays at zero. This injects linguistic knowledge into a statistical model and could transfer to other smoothed sequence models.
• I4. Locale-aware character classes: writing tokenizer rules with POSIX classes such as [[:alpha:]] instead of hard-coded alphabets makes one rule work across scripts. This is valuable when processing Cyrillic and Latin text in the same system.
• I5. Declarative PoS mapping to the sense inventory: a configurable mapping bridges the tagset to WordNet categories and can also look up the surface form. For example, falling can receive adjective senses as well as verb senses. A similar mapping layer could connect a Ukrainian tagset to external lexical resources.