Elamur Research
☰
home / papers / 2012 / langid.py: An Off-the-shelf Language Identification Tool

langid.py: An Off-the-shelf Language Identification Tool

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P12-3005.pdf

Summary

Language identification (LangID) underpins many pipelines, including web corpus construction, search, social media organisation and machine translation. Training a custom classifier for each application, however, requires costly labelled in-domain data. Lui and Baldwin address this with langid.py, a pre-trained tool designed to work across domains without user training. It uses a multinomial naive Bayes classifier over byte n-grams (n = 1–4). Features are chosen by LD feature selection, which favours n-grams that carry much Information Gain about language but little about domain. The embedded model was trained on the multi-domain corpus of Lui and Baldwin (2011), covering 97 languages from five domains. Feature extraction is a single pass through an Aho-Corasick automaton, and classification relies on numpy, so the tool fits in one file. The authors tested it on seven datasets that are independent of its training data, two of them from Twitter. langid.py kept accuracy consistently high, beat the other off-the-shelf tools on microblogs, and ran much faster than TextCat. It is therefore a practical default when no in-domain training data exists.

Key points

  • Compared tools: langid.py (97 languages), LangDetect (53), TextCat (75) and CLD (64+). The test sets were EuroGov, TCL, Wikipedia, EMEA, EuroParl, T-BE and T-SC.
  • langid.py accuracy ranged from 0.886 (T-SC) to 0.992 (EuroParl). Other results include 0.987 on EuroGov, 0.904 on TCL and 0.941 on T-BE.
  • CLD was the fastest system but less accurate on most sets, though it led on EMEA by +0.044. TextCat was 6–144× slower and less accurate everywhere.
  • On Twitter data, langid.py's 0.94 on T-BE is competitive with a specially tuned short-text method that scores 0.92–0.98. On T-SC it reaches 0.89 without restricting the candidate language set.
  • The package also ships LDfeatureselect.py and train.py for building custom models. It can be used from the command line, as a Python library, or as a WSGI-compliant web service that returns JSON.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Computing & GPUs

Cited by

264 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 255 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{lui2012langid,
  title = {{langid.py: An Off-the-shelf Language Identification Tool}},
  author = {Lui, Marco and Baldwin, Timothy},
  booktitle = {ACL 2012 System Demonstrations},
  year = {2012},
  url = {https://aclanthology.org/P12-3005/}
}

References

18 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alfred V. Aho and Margaret J. Corasick. 1975. Efficient string matching: an aid to bibliographic search. Communications of the ACM, 18(6):333–340, June. find
  2. Timothy Baldwin and Marco Lui. 2010. Language identification: The long and the short of the matter. In Proceedings of NAACL HLT 2010, pages 229–237, Los Angeles, USA. find
  3. Jamie Callan and Mark Hoy, 2009. ClueWeb09 Dataset. Available at http://boston.lti.cs. cmu.edu/Data/clueweb09/. ↗
  4. Simon Carter, Wouter Weerkamp, and Manos Tsagkias. to appear. Microblog language identification: Overcoming the limitations of short, unedited and idiomatic text. Language Resources and Evaluation Journal. find
  5. William B. Cavnar and John M. Trenkle. 1994. Ngram-based text categorization. In Proceedings of the Third Symposium on Document Analysis and Information Retrieval, Las Vegas, USA. find
  6. Hakan Ceylan and Yookyung Kim. 2009. Language identification of search engine queries. In Proceedings of ACL2009, pages 1066–1074, Singapore. find
  7. George Forman. 2003. An Extensive Empirical Study of Feature Selection Metrics for Text Classification. Journal of Machine Learning Research, 3(7-8):1289– 1305, October. find
  8. Rayid Ghani, Rosie Jones, and Dunja Mladenic. 2004. Building Minority Language Corpora by Learning to Generate Web Search Queries. Knowledge and Information Systems, 7(1):56–83, February. find
  9. Harald Hammarstrom. 2007. A Fine-Grained Model for Language Identication. In Proceedings of iNEWS07, pages 14–20. find
  10. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. MT summit, 11. find
  11. Marco Lui and Timothy Baldwin. 2011. Cross-domain feature selection for language identification. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 553–561, Chiang Mai, Thailand. find
  12. Andrew McCallum and Kamal Nigam. 1998. A comparison of event models for Naive Bayes text classification. In Proceedings of the AAAI-98 Workshop on Learning for Text Categorization, Madison, USA. find
  13. J.R. Quinlan. 1986. Induction of Decision Trees. Machine Learning, 1(1):81–106, October. find
  14. Penelope Sibun and Jeffrey C. Reynar. 1996. Language determination: Examining the issues. In Proceedings of the 5th Annual Symposium on Document Analysis and Information Retrieval, pages 125–135, Las Vegas, USA. find
  15. Jörg Tiedemann. 2009. News from OPUS - A Collection of Multilingual Parallel Corpora with Tools and Interfaces. Recent Advances in Natural Language Processing, V:237–248. find
  16. Erik Tromp and Mykola Pechenizkiy. 2011. Graph-Based N-gram Language Identification on Short Texts. In Proceedings of Benelearn 2011, pages 27–35, The Hague, Netherlands. find
  17. Tommi Vatanen, Jaakko J. Vayrynen, and Sami Virpioja. 2010. Language identification of short text segments with n-gram models. In Proceedings of LREC 2010, pages 3423–3430. find
  18. Yiming Yang and Jan O. Pedersen. 1997. A comparative study on feature selection in text categorization. In Proceedings of ICML 97. find