Elamur Research⌕
☰
home / papers / 2026 / CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

Paper · 2026 · Pedro Ortiz Suarez, Laurie Burchell, Catherine Arnett et al.

Summary

Language identification (LID) is often treated as a solved task. In practice, it remains unreliable for many languages on noisy, mixed web text, which is the main source of multilingual pre-training corpora. Most existing benchmarks, such as FLORES, UDHR and Bible data, are clean, translated or religious, so they give an overly optimistic picture. The authors built CommonLID, an openly licensed web-domain benchmark. Native speakers, mostly recruited from the NLP community through hackathons, labelled text line by line. The source material was sampled from Common Crawl and MADLAD-400. With this benchmark and five other evaluation sets, they compared eight widely used LID models and also tested several OpenAI GPT models. No system works well across every domain. GlotLID and CLD2 offer the best overall trade-off between coverage and speed, but common evaluations overstate accuracy on web data, especially for low-resource languages. The resource helps practitioners choose and build LID tools suited to curating web corpora for under-served languages.

Key points

  • CommonLID covers 109 language varieties, 78 of which have at least 100 annotated lines. It was produced with more than eighty annotators and is released on Hugging Face (commoncrawl/CommonLID) together with its code.
  • Candidate documents came from the WET files of CC-MAIN-2024-22 and CC-MAIN-2025-05, selected via the Ungoliant/OSCAR pipeline. Selection used fastText, OpenLID and GlotLID, with additional documents from MADLAD-400. The authors note that relying on existing LID models for selection biases which data gets included.
  • No model exceeds 75% F1 on all evaluation datasets. GlotLID scores highest on FLORES and Bibles, partly because Bible text is often its only training data for long-tail languages.
  • On a shared core set of 76 languages, GlotLID reaches 93.5 F1 against 91.8 for GPT-5. On the 294-language Afro* subset the gap widens to 90.6 versus 66.6, despite the much higher cost of running GPT models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations🎓Massachusetts Institute of TechnologyCambridge · United States211 works · 69,000 citations🏢IBMArmonk · United States139 works · 70,691 citations🎓University of North Carolina at Chapel HillChapel Hill · United States67 works · 16,474 citations🏛️National Research Council CanadaOttawa · Canada59 works · 21,677 citations🔬Institut national de recherche en sciences et technologies du numériqueLe Chesnay-Rocquencourt · France84 works · 25,414 citations🔬German Research Centre for Artificial IntelligenceKaiserslautern · Germany77 works · 29,479 citations🎓University of ManchesterManchester · United Kingdom87 works · 44,938 citations🎓Monash UniversityMelbourne · Australia60 works · 54,866 citations🎓École Polytechnique Fédérale de LausanneLausanne · Switzerland44 works · 7,050 citations🎓Imperial College LondonLondon · United Kingdom48 works · 12,181 citations🎓University of MannheimMannheim · Germany29 works · 8,347 citations🎓Universität HamburgHamburg · Germany79 works · 5,422 citations🎓IT University of CopenhagenCopenhagen · Denmark27 works · 2,406 citations🎓Michigan State UniversityEast Lansing · United States23 works · 1,077 citations🎓King Saud UniversityRiyadh · Saudi Arabia37 works · 527 citations🎓University of BucharestBucharest · Romania21 works · 516 citations🔬Hasso Plattner InstitutePotsdam · Germany12 works · 4,998 citations🎓Benha UniversityBanhā · Egypt14 works · 89 citations🎓University of PretoriaPretoria · South Africa9 works · 755 citations🎓University of IndonesiaDepok · Indonesia8 works · 589 citations🎓Stevens Institute of TechnologyHoboken · United States10 works · 95 citations🎓Monash University, IndonesiaTangerang · Indonesia5 works · 160 citations🎓University of New HavenWest Haven · United States2 works · 17 citations🏢NEC (Germany)Heidelberg · Germany3 works · 114 citations🎓Khatam UniversityTehran · Iran4 works · 31 citations🎓Universität TrierTrier · Germany8 works · 9 citations🎓National UniversityManila · Philippines5 works · 353 citations🎓National Taiwan University of Science and TechnologyTaipei · Taiwan4 works · 6 citations🎓Universidad de ZaragozaZaragoza · Spain5 works · 5 citations🎓University of BathBath · United Kingdom5 works · 351 citations

Also on the byline: Pelita Harapan University · Thammasat University · Binus University · University of Ha'il · Wad Medani Ahlia University · University of the People.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Pedro Ortiz Suarez, Laurie Burchell, Catherine Arnett, Rafael Mosquera, Sara Hincapié Monsalve, Thom Vaughan, Damian Stewart, Malte Ostendorff, Idris Abdulmumin, Vukosi Marivate, Shamsuddeen Hassan Muhammad, Atnafu Lambebo Tonja, Hend Al-Khalifa, Nadia Ghezaiel Hammouda, Verrah Akinyi Otiende, Tack Hwa Wong, Jakhongir Saydaliev, Melika Nobakhtian, Muhammad Ravi Shulthan Habibi, Chalamalasetti Kranti, Carol Muchemi, Khang Nguyen, Faisal Muhammad Adam, Luis Frentzen Salim, Reem Alqifari, Cynthia Jayne Amol, Joseph Marvin Imperial, Ilker Kesen, Ahmad Mustafid, Pavel Stepachev, Leshem Choshen, David Anugraha, Hamada Nayel, Seid Muhie Yimam, Vallerie Alexandra Putra, My Chiffon Nguyen, Azmine Toushik Wasi, Gouthami Vadithya, Rob van der Goot, Lanwenn ar C'horr, Karan Dua, Andrew Yates, Mithil Bangera, Yeshil Bangera, Hitesh Laxmichand Patel, Shu Okabe, Fenal Ashokbhai Ilasariya, Dmitry Gaynullin, Genta Indra Winata, Yiyuan Li, Juan Pablo Martínez, Amit Agarwal, Ikhlasul Akmal Hanif, Raia Abu Ahmad, Esther Adenuga, Filbert Aurelian Tjiaranata, Weerayut Buaphet, Michael Anugraha, Sowmya Vajjala, Benjamin L Rice, Azril Hafizi Amirudin, Jesujoba Oluwadara Alabi, Srikant Panda, Yassine Toughrai, Bruhan Kyomuhendo, Daniel Ruffinelli, Akshata, Manuel Goulão, Ej Zhou, Ingrid Gabriela Franco Ramirez, Cristina Aggazzotti, Konstantin Dobler, Jun Kevin, Quentin Pagès, Nicholas Andrews, Nuhu Ibrahim, Mattes Ruckdeschel, Amr Keleg, Mike Zhang, Casper Rufaro Muziri, Saron Samuel, Sotaro Takeshita, Kun Kerdthaisong, Luca Foppiano, Rasul Dent, Tommaso Green, Ahmad Mustapha Wali, Kamohelo Makaaka, Vicky Feliren, Inshirah Idris, Hande Celikkanat, Abdulhamid Abubakar, Jean Maillard, Benoît Sagot, Thibault Clérice, Kenton Murray and Sarah K. K. Luger. 2026. CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1527

@inproceedings{suarez2026commonlid,
  title = {{CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data}},
  author = {Pedro Ortiz Suarez, Laurie Burchell, Catherine Arnett, Rafael Mosquera, Sara Hincapié Monsalve, Thom Vaughan, Damian Stewart, Malte Ostendorff, Idris Abdulmumin, Vukosi Marivate, Shamsuddeen Hassan Muhammad, Atnafu Lambebo Tonja, Hend Al-Khalifa, Nadia Ghezaiel Hammouda, Verrah Akinyi Otiende, Tack Hwa Wong, Jakhongir Saydaliev, Melika Nobakhtian, Muhammad Ravi Shulthan Habibi, Chalamalasetti Kranti, Carol Muchemi, Khang Nguyen, Faisal Muhammad Adam, Luis Frentzen Salim, Reem Alqifari, Cynthia Jayne Amol, Joseph Marvin Imperial, Ilker Kesen, Ahmad Mustafid, Pavel Stepachev, Leshem Choshen, David Anugraha, Hamada Nayel, Seid Muhie Yimam, Vallerie Alexandra Putra, My Chiffon Nguyen, Azmine Toushik Wasi, Gouthami Vadithya, Rob {van der Goot}, Lanwenn ar C'horr, Karan Dua, Andrew Yates, Mithil Bangera, Yeshil Bangera, Hitesh Laxmichand Patel, Shu Okabe, Fenal Ashokbhai Ilasariya, Dmitry Gaynullin, Genta Indra Winata, Yiyuan Li, Juan Pablo Martínez, Amit Agarwal, Ikhlasul Akmal Hanif, Raia Abu Ahmad, Esther Adenuga, Filbert Aurelian Tjiaranata, Weerayut Buaphet, Michael Anugraha, Sowmya Vajjala, Benjamin L Rice, Azril Hafizi Amirudin, Jesujoba Oluwadara Alabi, Srikant Panda, Yassine Toughrai, Bruhan Kyomuhendo, Daniel Ruffinelli, Akshata, Manuel Goulão, Ej Zhou, Ingrid Gabriela Franco Ramirez, Cristina Aggazzotti, Konstantin Dobler, Jun Kevin, Quentin Pagès, Nicholas Andrews, Nuhu Ibrahim, Mattes Ruckdeschel, Amr Keleg, Mike Zhang, Casper Rufaro Muziri, Saron Samuel, Sotaro Takeshita, Kun Kerdthaisong, Luca Foppiano, Rasul Dent, Tommaso Green, Ahmad Mustapha Wali, Kamohelo Makaaka, Vicky Feliren, Inshirah Idris, Hande Celikkanat, Abdulhamid Abubakar, Jean Maillard, Benoît Sagot, Thibault Clérice, Kenton Murray and Sarah K. K. Luger},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1527},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1527}
}

References

49 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022. Towards a cleaner document-oriented multilingual crawled corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4344–4355, Marseille, France. European Language Resources Association. http…
  2. Julien Abadji, Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2021. Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus. In Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-9) 2021. Limerick, 12 July 2021 (… find
  3. Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Alcides Inciarte. 2022. AfroLID: A neural language identification tool for African languages. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1958–1981, Abu Dhabi, United Arab Emirates. Asso… find
  4. Željko Agić and Ivan Vulić. 2019. JW300: A wide-coverage parallel corpus for low-resource languages. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3204– 3210, Florence, Italy. Association for Computational Linguistics.
  5. Verena Blaschke, Hinrich Schuetze, and Barbara Plank. 2023. A survey of corpora for Germanic low-resource languages and dialects. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 392–414, Tórshavn, Faroe Islands. University of Tartu Library.
  6. Su Lin Blodgett, Johnny Wei, and Brendan O’Connor. 2017. A dataset and classifier for recognizing social media English. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 56–61, Copenhagen, Denmark. Association for Computational Linguistics.
  7. Ralf Brown. 2014. Non-linear mapping for improved identification of 1300+ languages. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 627– 632, Doha, Qatar. Association for Computational Linguistics. find
  8. Laurie Burchell, Alexandra Birch, Nikolay Bogoychev, and Kenneth Heafield. 2023. An open dataset and model for language identification. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 865–879, Toronto, Canada. Association for…
  9. Laurie Burchell, Alexandra Birch, Robert Thompson, and Kenneth Heafield. 2024a. Code-switched language identification is harder than you think. In Proceedings of the 18th Conference of the European 33072 10 find
  10. Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 646–658, St. Julian’s, Malta. Association for Computational Linguistics. find
  11. Laurie Burchell, Jean Maillard, Antonios Anastasopoulos, Christian Federmann, Philipp Koehn, and Skyler Wang. 2024b. Findings of the WMT 2024 shared task of the open language data initiative. In Proceedings of the Ninth Conference on Machine Translation, pages 110–117, Miami, Florida, USA. Associati… find
  12. Laurie Vear Burchell. 2024. Improving natural language processing for under-served languages through increased training data diversity. Ph.D. thesis, University of Edinburgh. find
  13. Tommaso Caselli, Roberto Cibin, Costanza Conforti, Enrique Encinas, and Maurizio Teli. 2021. Guiding principles for participatory design-inspired natural language processing. In Proceedings of the 1st Workshop on NLP for Positive Impact, pages 27–35, Online. Association for Computational Linguistics… find
  14. Jonathan Dunn. 2020. Mapping languages: the corpus of global language use. Lang. Resour. Eval., 54(4):999–1018. find
  15. Negar Foroutan, Jakhongir Saydaliev, Ye Eun Kim, and Antoine Bosselut. 2025. Conlid: Supervised contrastive learning for low-resource language identification. arXiv preprint arXiv:2506.15304. find
  16. Karën Fort, Gilles Adda, and K. Bretonnel Cohen. 2011. Last words: Amazon Mechanical Turk: Gold mine or coal mine? Computational Linguistics, 37(2):413– 420. find
  17. Karën Fort, Gilles Adda, Benoît Sagot, Joseph Mariani, and Alain Couillault. 2014. Crowdsourcing for language resource development: Criticisms about amazon mechanical turk overpowering use. In Human Language Technology Challenges for Computer Science and Linguistics, pages 303–314, Cham. Springer In… find
  18. Isaac Caswell. 2024. Funlangid. https://github.com/google-research/url-nlp/ tree/main/fun_langid. ↗
  19. Rob Van Der Goot. 2025. Identifying open challenges in language identification. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18207–18227, Vienna, Austria. Association for Computational Linguistics. find
  20. Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna. 2020. Language ID in the wild: Unexpected challenges on the path to a thousand-language web text corpus. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6588–6608, Barcelona, Spain (Online). Inter…
  21. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computat… find
  22. Isaac Caswell, Elizabeth Nielsen, Jiaming Luo, Colin Cherry, Geza Kovacs, Hadar Shemtov, Partha Talukdar, Dinesh Tewari, Baba Mamadi Diane, Djib-rila Diane, Solo Farabado Cissé, Koulako Moussa Doumbouya, Edoardo Ferrante, Alessandro Guasoni, Christopher Homan, Mamadou K. Keita, Sudhamoy DebBarma, Al… find
  23. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. The FLORES evaluation datasets for low-resource machine translation: Nepali–English and Sinhala– English. In Proceedings of the 2019 Conference on Empirical Metho…
  24. Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, and 1 others. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. find
  25. David Dale, Laurie Burchell, Jean Maillard, Idris Abdulmumin, Antonios Anastasopoulos, Isaac Caswell, and Philipp Koehn. 2025. Findings of the WMT 2025 shared task of the open language data initiative. In Proceedings of the Tenth Conference on Machine Translation, pages 495–502, Suzhou, China. Assoc… find
  26. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. FastText.zip: Compressing text classification models. arXiv e-prints, arXiv:1612.03651.
  27. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain. Associatio… find
  28. Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schuetze. 2023. GlotLID: Language 33073 11 identification for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6155–6218, Singapore. Association for Computational Linguistics. find
  29. Amr Keleg, Sharon Goldwater, and Walid Magdy. 2023. ALDi: Quantifying the Arabic level of dialectness of text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10597–10611, Singapore. Association for Computational Linguistics.
  30. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, and 1 others. 2023. Dspy: Compiling declarative language model calls into self-improving pipelines. CoRR.
  31. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dyna… find
  32. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79–86, Phuket, Thailand. find
  33. Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro… — Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
  34. Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2023. Madlad-400: A multilingual and document-level large audited dataset. In Advances in Neural Information Processing Systems, volume 36, pages 67284–67296. Curran As…
  35. Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole Hunter, Kevin Klyman, Christopher Klamm, Hailey Schoelkopf, Nikhil Singh, Manuel Cherep, Ahmad Mustafa Anis, An Dinh, Caroline Chitongo, Da Yin, and 30 oth… find
  36. Thomas Mayer and Michael Cysouw. 2014. Creating a massively parallel Bible corpus. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 3158– 3163, Reykjavik, Iceland. European Language Resources Association (ELRA).
  37. Paul McNamee. 2005. Language identification: a solved problem suitable for undergraduate instruction. J. Comput. Sci. Coll., 20(3):94–101. find
  38. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth I…
  39. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language Resources and Eval…
  40. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, and 20 o… find
  41. Chijioke I Okorie. 2025. The fair dealing/fair use landscape for artificial intelligence innovation and computational research in africa. International Review of Law, Computers & Technology, pages 1–28. find
  42. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. In Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 – 16, Mannh… find
  43. Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. 2025. Fineweb2: One pipeline to scale them all — adapting pre-training data processing to every language. In Second Conference o…
  44. Alex Salcianu, Andy Golding, Anton Bakalov, Chris Alberti, Daniel Andor, David Weiss, Emily Pitler, Greg Coppola, Jason Riesa, Kuzman Ganchev, Michael Ringgaard, Nan Hua, Ryan McDonald, Slav Petrov, 33074 12 find
  45. Stefan Istrate, and Terry Koo. 2020. cld3: Compact language detector v3. https://github.com/ google/cld3. Archived; Apache-2.0 License. ↗
  46. Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021. CCMatrix: Mining billions of high-quality parallel sentences on the web. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Co…
  47. Dick Sites, Jason Riesa, and Ivan Giuliani. 2013. cld2: Compact language detector 2. https://github.com/CLD2Owners/cld2. Apache-2.0 License; detects 83 languages :contentRefer-ence[oaicite:0]index=0. ↗
  48. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey. European Language Resources Association (ELRA). of North Carolina at Chapel Hill, 48 Academia Aragonesa de la Leng… find
  49. Arkaitz Zubiaga, Iñaki San Vicente, Pablo Gamallo, José Ramom Pichel, Inaki Alegria, Nora Aranberri, Aitzol Ezeiza, and Víctor Fresno. 2016. Tweetlid: a benchmark for tweet language identification. Language Resources and Evaluation, 50(4):729–766. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/suarez-2026-commonlid-re-evaluating-state-art/