Elamur Research⌕
☰
home / papers / 2014 / Automatic Detection and Language Identification of Multilingual Documents

Automatic Detection and Language Identification of Multilingual Documents

Summary

Most language identification tools assume each document is written in exactly one language. This paper drops that assumption. It tackles documents that mix several languages, aiming to flag them as multilingual, name the languages involved and estimate how much text belongs to each. The authors frame this as multi-label classification. Each document is represented as counts of byte n-grams chosen by Information Gain, following earlier monolingual work by Lui and Baldwin (2011). A generative mixture model, borrowed from supervised topic modeling, treats the document as tokens drawn from an unknown blend of known languages. A Gibbs sampler assigns tokens to languages, and the system picks the language set with the highest posterior probability. Training needs only monolingual data, which matters because no standard multilingual corpora exist. On synthetic Wikipedia-based data and on real web pages from low-density language crawls, the method beats LINGUINI and SegLang and gives accurate proportion estimates. This makes it useful for cleaning NLP input, collecting minority-language corpora and mining parallel text.

Key points

  • Data: Experiments start with the ALTW 2010 shared task data (mono- and bilingual Wikipedia documents). The authors then build a synthetic dataset, WikipediaMulti, and release it publicly.
  • Proportion estimates: With emission rate correction, predicted language proportions closely match the true ones (Pearson's r 0.981, MAE 0.0241). Without the correction, results are much weaker (r 0.863, MAE 0.108).
  • Real-world test, data: The authors use 149 web documents covering 42 languages, provided by Ben King from targeted low-density language crawls (King and Abney, 2013). Models are trained on roughly 20–80 kB per language, mostly from the UDHR, with Wikipedia or Bible text where UDHR was missing.
  • Real-world test, results: All systems detected the target language in every document. The proposed method achieved the best F-score for spotting English inclusions, producing fewer false positives than LINGUINI and SegLang. Its misses occurred mainly when the English portion was small.
  • Code and future work: The reference implementation is available at github.com/saffsd/polyglot. The authors list several open directions: tuning α/β, cross-domain robustness and handling unknown languages via non-parametric models such as Hierarchical Dirichlet Processes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing

Cited by

30 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 29 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00163

@article{lui2014automatic,
  title = {{Automatic Detection and Language Identification of Multilingual Documents}},
  author = {Lui, Marco and Lau, Jey Han and Baldwin, Timothy},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2014},
  doi = {10.1162/tacl\_a\_00163},
  url = {https://doi.org/10.1162/tacl_a_00163}
}

References

42 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Steven Abney and Steven Bird. 2010. The human language project: building a universal corpus of the world’s languages. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 88–97. Association for Computational Linguistics. find
  2. Beatrice Alex, Amit Dubey, and Frank Keller. 2007. Using foreign inclusion detection to improve parsing performance. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning 2007 (EMNLP-CoNLL 2007), pages 151–160, Prague,… find
  3. Timothy Baldwin and Marco Lui. 2010a. Language identification: The long and the short of the matter. In Proceedings of Human Language Technologies: The 11th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL HLT 2010), pages 229–237, Los Angeles,… find
  4. Timothy Baldwin and Marco Lui. 2010b. Multilingual language identification: ALTW 2010 shared task dataset. In Proceedings of the Australasian Language Technology Workshop 2010 (ALTW 2010), pages 5–7, Melbourne, Australia. find
  5. Shane Bergsma, Paul McNamee, Mossaab Bagdouri, Clayton Fink, and Theresa Wilson. 2012. Language identification for creating language-specific Twitter collections. In Proceedings the Second Workshop on Language in Social Media (LSM2012), pages 65–74, Montréal, Canada. find
  6. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993–1022. find our card
  7. Alessio Bosca and Luca Dini. 2010. Language identification strategies for cross language information retrieval. In Working Notes of the Cross Language Evaluation Forum (CLEF). find
  8. Jamie Callan and Mark Hoy, 2009. ClueWeb09 Dataset. Available at http://boston.lti.cs.cmu.edu/Data/clueweb09/. ↗
  9. William B. Cavnar and John M. Trenkle. 1994. Ngram-based text categorization. In Proceedings of the Third Symposium on Document Analysis and Information Retrieval, pages 161–175, Las Vegas, USA. find
  10. Hakan Ceylan and Yookyung Kim. 2009. Language identification of search engine queries. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 1066–1074, Singapore. find
  11. Paul Cook and Marco Lui. 2012. langid.py for better language modelling. In Proceedings of the Australasian Language Technology Association Workshop 2012, pages 107–112, Dunedin, New Zealand. Rafael Dueire Lins and Paulo Gonçalves. 2004. Automatic language identification of written texts. In Proceed… find
  12. Ted Dunning. 1994. Statistical identification of language. Technical Report MCCS 940-273, Computing Research Laboratory, New Mexico State University. find
  13. Rayid Ghani, Rosie Jones, and Dunja Mladenic. 2004. Building minority language corpora by learning to generate web search queries. Knowledge and Information Systems, 7(1):56–83. find
  14. Emmanuel Giguet. 1995. Categorisation according to language: A step toward combining linguistic knowledge and statistical learning. In Proceedings of the 4th International Workshop on Parsing Technologies (IWPT-1995), Prague, Czech Republic. find
  15. Gregory Grefenstette. 1995. Comparing two language identification schemes. In Proceedings of Analisi Statistica dei Dati Testuali (JADT), pages 263–268, Rome, Italy. find
  16. Thomas L. Griffiths and Mark Steyvers. 2004. Finding scientific topics. Proceedings of the National Academy of Sciences, 101:5228–5235. find
  17. Thomas Griffiths. 2002. Gibbs sampling in the generative model of latent Dirichlet allocation. find
  18. Technical Report, Stanford University. Baden Hughes, Timothy Baldwin, Steven Bird, Jeremy Nicholson, and Andrew MacKinlay. 2006. Reconsidering language identification for written language resources. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC 2006),… find
  19. Genitiro Kikui. 1996. Identifying the coding system and language of on-line documents on the internet. In Proceedings of the 16th International Conference on Computational Linguistics (COLING ’96), pages 652– 657, Kyoto, Japan. find
  20. Ben King and Steven Abney. 2013. Labeling the languages of words in mixed-language documents using weakly supervised methods. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1110–1119, Atlanta, G… find
  21. Canasai Kruengkrai, Prapass Srichaivattana, Virach Sornlertlamvanich, and Hitoshi Isahara. 2005. Language identification based on string kernels. In Proceedings of the 5th International Symposium on Communications and Information Technologies (ISCIT-2005), pages 896–899, Beijing, China. find
  22. David D. Lewis. 1997. The Reuters-21578 data set. available at http://www.daviddlewis.com/resources/testcollections/ reuters21578/. ↗
  23. Wang Ling, Guang Xiang, Chris Dyer, Alan Black, and Isabel Trancoso. 2013. Microblogs as parallel corpora. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 176–186, Sofia, Bulgaria, August. Association for Computational Linguis… find
  24. Jicheng Liu and Chunyan Liang. 2008. Text Categorization of Multilingual Web Pages in Specific Domain. In Proceedings of the 12th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining, PAKDD’08, pages 938–944, Osaka, Japan. find
  25. Marco Lui and Timothy Baldwin. 2011. Cross-domain feature selection for language identification. In Proceedings of the 5th International Joint Conference on Natural Language Processing (IJCNLP 2011), pages 553–561, Chiang Mai, Thailand. Bruno Martins and Mário J. Silva. 2005. Language identificatio… find
  26. Andrew McCallum and Kamal Nigam. 1998. A comparison of event models for Naive Bayes text classification. In Proceedings of the AAAI-98 Workshop on Learning for Text Categorization, pages Available as Technical Report WS–98–05, AAAI Press., Madison, USA. find
  27. Andrew Kachites McCallum. 1999. Multi-label text classification with a mixture model trained by EM. In Proceedings of AAAI 99 Workshop on Text Learning. find
  28. Paul McNamee. 2005. Language identification: a solved problem suitable for undergraduate instruction. Journal of Computing Sciences in Colleges, 20(3):94–101. find
  29. Jian-Yun Nie, Michel Simard, Pierre Isabelle, and Richard Durand. 1999. Cross-language information retrieval based on parallel texts and automatic mining of parallel texts from the web. In Proceedings of 22nd International ACM-SIGIR Conference on Research and Development in Information Retrieval (SI… find
  30. John M. Prager. 1999a. Linguini: language identification for multilingual documents. In Proceedings the 32nd Annual Hawaii International Conference on Systems Sciences (HICSS-32), Maui, Hawaii. find
  31. John M. Prager. 1999b. Linguini: Language identification for multilingual documents. Journal of Management Information Systems, 16(3):71–101. find
  32. John Ross Quinlan. 1993. C4.5: Programs for Machine Learning. find our card
  33. Morgan Kaufmann, San Mateo, USA. Daniel Ramage, David Hall, Ramesh Nallapati, and Christopher D. Manning. 2009. Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP 2009… find
  34. Radim Rehurek and Milan Kolkus. 2009. Language Identification on the Web: Extending the Dictionary Method. In Proceedings of Computational Linguistics and Intelligent Text Processing, 10th International Conference (CICLing 2009), pages 357–368, Mexico City, Mexico. find
  35. Philip Resnik. 1999. Mining the Web for bilingual text. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 527–534, College Park, USA. find
  36. Kevin P Scannell. 2007. The Crúbadán Project: Corpus building for under-resourced languages. In Building and Exploring Web Corpora: Proceedings of the 3rd Web as Corpus Workshop, pages 5–15, Louvain-la-Neuve, Belgium. find
  37. W. J. Teahan. 2000. Text Classification and Segmentation Using Minimum Cross-Entropy. In Proceedings the 6th International Conference “Recherche d’Information Assistee par Ordinateur” (RIAO’00), pages 943–961, Paris, France. find
  38. Yee Whye Teh, Michael I. Jordan, Matthew J. Beal, and David M. Blei. 2006. Hierarchical Dirichlet processes. Journal of the American Statistical Association, 101:1566–1581. Jörg Tiedemann and Nikola Ljubešić. 2012. Efficient discrimination between closely related languages. In Proceedings of the… find
  39. Giang Binh Tran, Dat Ba Nguyen, and Bin Thanh Kieu. 2010. N-gram based approach for multilingual language identification. poster. available at http://comp.mq.edu.au/programming/ task_description/VILangTek.pdf. ↗
  40. Fei Xia, Carrie Lewis, and William D. Lewis. 2010. Language ID for a thousand languages. In LSA Annual Meeting Extended Abstracts, Baltimore,USA. find
  41. Hiroshi Yamaguchi and Kumiko Tanaka-Ishii. 2012. Text segmentation by language using minimum description length. In Proceedings the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 969–978, Jeju Island, Korea. find
  42. Alexander Yeh. 2000. More accurate tests for the statistical significance of result differences. In Proceedings of the 18th International Conference on Computational Linguistics (COLING 2000), pages 947–953, Saarbrücken, Germany. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lui-2014-automatic-detection-language-identification-multilingual/