Elamur Research⌕
☰
home / papers / 2026 / Do LLMs Judge Distantly Supervised Named Entity Labels Well? Constructing the JudgeWEL Dataset

Do LLMs Judge Distantly Supervised Named Entity Labels Well? Constructing the JudgeWEL Dataset

Summary

Luxembourgish, with roughly 400,000 speakers, has very few labelled NLP datasets. Its only prior NER set, RTL-NER (Lothritz et al., 2022), contains about 5,500 manually tagged news sentences. The authors propose a largely automatic method for building a bigger one. Their pipeline treats hyperlinks in Luxembourgish Wikipedia articles as distant supervision and assigns entity types by looking up the linked items in Wikidata, which yields BIO tags. Link-based labels are often incomplete or wrong, so missing tags are added with LuxGPT-NER and regular expressions. Several LLMs then act as judges and decide whether to keep or discard each sentence. A 500-sentence human-checked sample serves as the reference for comparing the judges. The resulting dataset, judgeWEL, has 28,866 sentences, about five times the size of RTL-NER, with more balanced entity coverage. The open pipeline offers a reusable recipe for other under-resourced languages. (The text we received omitted the results sections, so the best-performing judge and the NER scores are not covered here.)

Key points

  • Source data: the Luxembourgish Wikipedia dump of 2024-03-29 (~77k articles), extracted with WikiExtractor. Entity types come from matching internal links to Wikidata entries.
  • Sentence selection: each article's intro is skipped, the next five sentences are taken, and short or redundant ones are filtered out.
  • Quality control: after LuxGPT-NER and regex corrections, multiple LLMs make a binary keep/discard judgement per sentence. A 500-sentence human check serves as the consensus baseline.
  • Resource: judgeWEL contains 28,866 sentences split 80/10/10 into Train/Dev/Test, about 5× larger than RTL-NER.
  • Entity types: the paper states five entity types, but the pipeline figure lists only PER, ORG, LOC and DATE. Check the full paper for the fifth type.
  • Availability: the pipeline and dataset are available at github.com/plumaj/judgeWEL.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Named entity recognition

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2s4piwjugsds

@inproceedings{plum2026do,
  title = {{Do LLMs Judge Distantly Supervised Named Entity Labels Well? Constructing the JudgeWEL Dataset}},
  author = {Plum, Alistair and Bernardy, Laura Maria and Ranasinghe, Tharindu},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2s4piwjugsds},
  url = {https://doi.org/10.63317/2s4piwjugsds}
}

References

66 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martine Adda-Decker, Thomas Pellegrini, Eric Bilinski, and Gilles Adda. 2008. Developments of “Lëtzebuergesch” Resources for Automatic Speech Processing and Linguistic Studies. In Proceedings of LREC. find
  2. David Ifeoluwa Adelani, Dana Ruiter, Monojit Choudhury, et al. 2022. Masakhaner 2.0: Africa-centric transfer learning for named entity recognition. In Proceedings of ACL. ↗
  3. Rami Al-Rfou, Vivek Kulkarni, Bryan Perozzi, and Steven Skiena. 2015. Polyglot-ner: Massive multilingual named entity recognition. In Proceedings of the SIAM International Conference on Data Mining. ↗
  4. Dhananjay Ashok and Zachary C Lipton. 2023. Promptner: Prompting for named entity recognition. arXiv preprint arXiv:2305.15444. ↗
  5. Laura Bernardy. 2022. A Luxembourgish GPT-2 Approach Based on Transfer Learning. Master’s thesis, University of Trier. find
  6. Tom B. Brown, Benjamin Mann, Nick Ryder, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33. ↗
  7. Siqi Chen, Yijie Pei, Zunwang Ke, and Wushour Silamu. 2021. Low-resource named entity recognition via the pre-training model. Symmetry, 13(5). ↗
  8. Yanru Chen, Yanan Zheng, and Zhilin Yang. 2023. Prompt-based metric learning for few-shot NER. In Findings of ACL. find
  9. Cohere for AI. 2025. Introducing command r+ and command a models. https://cohere.com/blog/command-a-command-r-plus. Accessed 2025. ↗
  10. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. ↗
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗
  12. Dimitra Farmakiotou, Vangelis Karkaletsis, John Koutsias, George Sigletos, Constantine D Spyropoulos, and Panagiotis Stamatopoulos. 2000. Rule-based named entity recognition for greek financial texts. In Proceedings of COMLEX 2000. find
  13. Xiaocheng Feng, Xiachong Feng, Bing Qin, Zhangyin Feng, and Ting Liu. 2018. Improving low resource named entity recognition using cross-lingual knowledge transfer. In Proceedings of IJCAI. find
  14. Daniela Gierschek. 2022. Detection of Sentiment in Luxembourgish User Comments. Ph.D. thesis, University of Luxembourg. find
  15. Peter Gilles. 2019. 39. Komplexe Überdachung II: Luxemburg. Die Genese Einer Neuen Nationalsprache. In Joachim Herrgen and Jürgen Erich Schmidt, editors, Sprache und Raum - Ein internationales Handbuch der Sprachvariation. Volume 4 Deutsch. De Gruyter Mouton, Berlin, Boston. find
  16. Hansi Hettiarachchi, Tharindu Ranasinghe, Paul Rayson, Ruslan Mitkov, Mohamed Gaber, Damith Premasiri, Fiona Anting Tan, and Lasitha Randunu Chandrakantha Uyangodage. 2025. Overview of the first workshop on language models for low-resource languages (LoResLM 2025). In Proceedings of LoResLM (COLING). ↗
  17. Y Hu, Q Chen, J Du, X Peng, VK Keloth, X Zuo, Y Zhou, Z Li, X Jiang, Z Lu, et al. 2024. Improving large language models for clinical named entity recognition via prompt engineering. JAMIA. ↗
  18. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. ↗
  19. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the nlp world. In Proceedings of ACL. find our card
  20. Gaurav Kamath and Sowmya Vajjala. 2025. Does synthetic data help named entity recognition for low-resource languages? arXiv preprint arXiv:2505.16814. ↗
  21. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In The Tenth Machine Translation Summit Proceedings of Conference. International Association for Machine Translation. find
  22. Thomas Lavergne, Gilles Adda, Martine Adda-Decker, and Lori Lamel. 2014. Automatic language identity tagging on word and sentence-level in multilingual text sources: a case-study on Luxembourgish. In Proceedings of LREC. find
  23. Jian Liu, Yufeng Chen, and Jinan Xu. 2022. Low-Resource NER by Data Augmentation With Prompting. In Proceedings of IJCAI. find
  24. Linlin Liu, Bosheng Ding, Lidong Bing, Shafiq Joty, Luo Si, and Chunyan Miao. 2021. MulDA: A multilingual data augmentation framework for low-resource cross-lingual NER. In Proceedings of ACL. find
  25. Cedric Lothritz, Bertrand Lebichot, Kevin Allix, Lisa Veiber, Tegawende Bissyande, Jacques Klein, Andrey Boytsov, Clément Lefebvre, and Anne Goujon. 2022. LuxemBERT: Simple and Practical Data Augmentation in Language Model Pre-Training for Luxembourgish. In Proceedings of LREC. find
  26. Anne-Marie Lutgen, Alistair Plum, Christoph Purschke, and Barbara Plank. 2025. Neural text normalization for Luxembourgish using real-life variation data. In Proceedings of VarDial (COL-ING). ↗
  27. Stephen Mayhew, Chen-Tse Tsai, and Dan Roth. 2017. Cheap translation for cross-lingual named entity recognition. In Proceedings of EMNLP. find
  28. Pablo N Mendes, Max Jakob, Andrés García-Silva, and Christian Bizer. 2011. Dbpedia spotlight: shedding light on the web of documents. In Proceedings of Semantic Systems. find
  29. Andrei Mikheev, Marc Moens, and Claire Grover. 1999. Named entity recognition without gazetteers. In Proceedings of EACL. find
  30. David Nadeau, Peter D Turney, and Stan Matwin. 2006. Unsupervised named-entity recognition: Generating gazetteers and resolving ambiguity. In Conference of the Canadian society for computational studies of intelligence. find
  31. OpenAI. 2025a. gpt-oss-120b & gpt-oss-20b model card. https://openai.com/index/gpt-oss-model-card/. ↗ our card
  32. OpenAI. 2025b. Introducing gpt-5. https://openai.com/index/introducing-gpt-5/. ↗
  33. OpenAI. 2025c. Introducing gpt-oss. https://openai.com/index/introducing-gpt-oss/. ↗
  34. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of ACL. find
  35. Fred Philippy, Shohreh Haddadan, and Siwen Guo. 2024. Forget NLI, Use a Dictionary: Zero-Shot Topic Classification for Low-Resource Languages with Application to Luxembourgish. In Proceedings of SIGUL (LREC-COLING). find
  36. Alistair Plum, Caroline Döhmer, Emilia Milano, Anne-Marie Lutgen, and Christoph Purschke. 2024. LuxBank: The first Universal Dependency treebank for Luxembourgish. In Proceedings of TLT. ↗
  37. Alistair Plum, Tharindu Ranasinghe, and Christoph Purschke. 2025. Text generation models for Luxembourgish with limited data: A balanced multilingual strategy. In Proceedings of VarDial (COL-ING). ↗
  38. Damith Premasiri, Tharindu Ranasinghe, Ruslan Mitkov, Mo El-Haj, and Ingo Frommholz. 2025. Survey on legal information extraction: current status and open challenges. Knowledge and Information Systems, 67(12):11287–11358. ↗
  39. Christoph Purschke. 2020. Attitudes Toward Multilingualism in Luxembourg. A Comparative Analysis of Online News Comments and Crowdsourced Questionnaire Data. Frontiers in AI, 3. find
  40. Lance Ramshaw and Mitch Marcus. 1995a. Text chunking using transformation-based learning. In Third Workshop on Very Large Corpora. find
  41. Lance Ramshaw and Mitch Marcus. 1995b. Text chunking using transformation-based learning. In Third Workshop on Very Large Corpora. find
  42. Tharindu Ranasinghe, Hansi Hettiarachchi, Nadeesha Chathurangi Naradde Vidana Pathirana, Damith Premasiri, Lasitha Uyangodage, Isuri Nanomi Arachchige, Alistair Plum, Paul Rayson, and Ruslan Mitkov. 2025. Sinhala encoder-only language models and evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8623–8636, Vienna, Austria. Association for Computational Linguistics. find
  43. Tharindu Ranasinghe, Alistair Plum, Christoph Purschke, and Marcos Zampieri. 2023. Publish or Hold? Automatic Comment Moderation in Luxembourgish News Articles. In Proceedings of RANLP. find
  44. Lisa F Rau. 1991. Extracting company names from text. In Proceedings of IEEE AIA. find
  45. Giuseppe Rizzo and Raphaël Troncy. 2012. Nerd: a framework for unifying named entity recognition and disambiguation extraction tools. In Proceedings of EACL. find
  46. Joshgun Sirajzade, Daniela Gierschek, and Christoph Schommer. 2020. An Annotation Framework for Luxembourgish Sentiment Analysis. In Proceedings of SLTU-CCURL (LREC). find
  47. Natalie D. Snoeren, Martine Adda-Decker, and Gilles Adda. 2010. The Study of Writing Variants in an Under-resourced Language: Some Evidence from Mobile N-Deletion in Luxembourgish. In Proceedings of LREC. find
  48. Beth M Sundheim. 1995. Overview of results of the muc-6 evaluation. In Sixth Message Understanding Conference (MUC-6). find
  49. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the 7th Conference on Natural Language Learning at HLT-NAACL 2003. ↗
  50. Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. ↗
  51. Hugo Touvron, Elena Voita, Gautier Izacard, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  52. Chen-Tse Tsai and Dan Roth. 2016. Cross-lingual wikification using multilingual embeddings. In Proceedings of NAACL-HLT. find
  53. Ahmet Üstün, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction finetuned open-access multilingual language model. In Proceedings of ACL. ↗ our card
  54. Ralph Weischedel, Sean Boisen, Daniel M Bikel, Robert Bobrow, Michael Crystal, William Ferguson, Allan Wechsler, PLUM Research Group, et al. 1996. Progress in information extraction. In TIPSTER TEXT PROGRAM PHASE II. find
  55. Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. Ontonotes: A large training corpus for enhanced processing. Handbook of Natural Language Processing and Machine Translation. Springer, 3(3):3–4. find
  56. Tingyu Xie, Qi Li, Jian Zhang, Yan Zhang, Zuozhu Liu, and Hongwei Wang. 2023. Empirical study of zero-shot NER with ChatGPT. In Proceedings of EMNLP. find
  57. Canwen Xu, Yichong Xu, Shuohang Wang, Yang Liu, Chenguang Zhu, and Julian McAuley. 2024. Small models are valuable plug-ins for large language models. In Findings of ACL. find
  58. Junjie Ye, Nuo Xu, Yikun Wang, Jie Zhou, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Llm-da: Data augmentation via large language models for few-shot named entity recognition. arXiv preprint arXiv:2402.14568. ↗
  59. Zhen Zhang, Yuhua Zhao, Hang Gao, and Mengting Hu. 2024. LinkNER: Linking Local Named Entity Recognition Models to Large Language Models using Uncertainty. In Proceedings of the ACM Web Conference. find
  60. Xingyu Zhu, Feifei Dai, Xiaoyan Gu, Bo Li, Meiou Zhang, and Weiping Wang. 2024. GL-NER: Generation-Aware Large Language Models for Few-Shot Named Entity Recognition. In Proceedings of ICANN. 10. find
  61. Language Resource References Giusepppe Attardi. 2015. Wikiextractor. https://github.com/attardi/wikiextractor. ↗
  62. Philipp Koehn and Josh Schroeder. 2005. sentence-splitter. https://github.com/mediacloud/sentence-splitter. ↗
  63. Cedric Lothritz, Bertrand Lebichot, Kevin Allix, Lisa Veiber, Tegawende Bissyande, Jacques Klein, Andrey Boytsov, Clément Lefebvre, and Anne Goujon. 2022. LuxemBERT: Simple and Practical Data Augmentation in Language Model Pre-Training for Luxembourgish. In Proceedings of LREC. find
  64. Hiroki Nakayama. 2018. seqeval: A Python framework for sequence labeling evaluation. https://github.com/chakki-works/seqeval. ↗
  65. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: A free collaborative knowledge base. Communications of the ACM, 57(10):78–85. find
  66. Wikipedia contributors. 2001. Wikipedia, the free encyclopedia. https://www.wikipedia.org/. Online resource, ongoing project. Wikimedia Foundation. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/plum-2026-do-llms-judge-distantly-supervised/