Elamur Research⌕
☰
home / papers / 2025 / SLENDER: Structured Outputs for SLM-based NER in Low-Resource Englishes

SLENDER: Structured Outputs for SLM-based NER in Low-Resource Englishes

Paper · 2025 · Nicole Ren and James Teo

Summary

Most NER systems are trained on American or British English and lose accuracy on English as written in other parts of the world. Proprietary LLMs handle this well through in-context learning, but they are costly and raise privacy issues. Small open models are cheaper but often produce malformed or invented output. The authors pair small language models with a compact output format called SLENDER. In this format each entity is written as entity|type, and entities are separated by semicolons. They combine it with constrained decoding and a carefully structured prompt. Experiments on the Worldwide dataset cover newswire from five low-resource regions and compare 0-, 3- and 5-shot setups. On average, SLENDER cuts inference time to about a third of what a full JSON format needs. Zero-shot Gemma-2-9B-it with SLENDER and constrained decoding beats spaCy's en_core_web_trf in F1 for every region. The authors also release a relabelled Worldwide test set. The work shows that output-format design is a cheap, overlooked way to make small models practical for structured extraction.

Key points

  • Data: the Worldwide dataset (Shan et al., 2023) has 9 labels and spans five regions: Indigenous Commonwealth, the Middle East, Latin America, Africa and Asia. It was preprocessed with Stanza.
    • The authors manually corrected test labels that broke the dataset's own guidelines. For example, "Allah" had been tagged PERSON, and "Covid-19" appeared as both DATE and MISCELLANEOUS.
  • Baselines: the format baseline is a complete JSON object with all 9 classes as keys. The model baseline is spaCy en_core_web_trf, with its OntoNotes labels mapped to the Worldwide classes.
    • Constraints were enforced with LMFE for JSON and Guidance for SLENDER.
  • Main results:
    • SLENDER gives roughly a 3x reduction in both average inference time and generated tokens, measured without constraints.
    • It is the better format on six of nine entity types for Gemma-2-9B-it.
    • spaCy remains stronger on Person and NORP.
  • When constraints help: constrained decoding helped SLENDER mostly in zero-shot, and its benefit shrank once examples were shown. For JSON it brought little gain and sometimes hurt.
  • Few-shot variance: demonstrations were one null example plus random training samples, drawn with 3 seeds. Only 8.10% of 864 few-shot results were identical across seeds, and the largest drop was 36.25%. All experiments ran on NVIDIA T4 GPUs.

Interesting ideas

  • I1. Spending structural tokens is a real cost: a flat format with just two reserved separators conveys the same entity–type pairs as nested JSON at a third of the generation time, though it drops entity positions.
  • I2. Adding "residual bin" classes (e.g. Food) that the task does not need catches frequent false positives that would otherwise pollute a catch-all class like Miscellaneous.
  • I3. Constrained decoding helps most when the output format is unfamiliar to the model and can hurt when it forces a familiar format away from learnt habits, so it should be tested per format rather than assumed beneficial.
  • I4. Few-shot prompts include a deliberate null example (text with no entities) and are scored over several random seeds, exposing how unstable results are with respect to example choice.
  • I5. Before trusting gold labels, the authors checked them against the dataset's own written definitions and fixed systematic inconsistencies, then released the cleaned split.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): LLM teacher tiers could return silver annotations (roles, entity types, attachments) as compact delimited strings rather than JSON, making large silver batches cheaper to produce.
  • I2 (world): when typing entities against the ontology, extra decoy categories could absorb common confusions so that the miscellaneous bucket stays meaningful and errors are easier to explain.
  • I3 (learn): when an LLM teacher must emit a custom format (e.g. frame or CoNLL-U-like lines), compare grammar-constrained and free generation before adopting either.
  • I4 (learn): teacher prompts could always carry a negative example, and evaluations could report spread across demonstration seeds as a built-in control.
  • I5 (tree): a similar guideline audit of UD treebanks or converted dialects could catch label noise before it reaches the perceptron parser or gparse training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Government Technology Agency.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,379 worksNamed entity recognition · 214 worksLow-resource languages · 287 works

Cite

Cite as Nicole Ren and James Teo. 2025. SLENDER: Structured Outputs for SLM-based NER in Low-Resource Englishes. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-industry.59

@inproceedings{ren2025slender,
  title = {{SLENDER: Structured Outputs for SLM-based NER in Low-Resource Englishes}},
  author = {Nicole Ren and James Teo},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-industry.59},
  url = {https://doi.org/10.18653/v1/2025.acl-industry.59}
}

References

18 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
  2. Gemma Team, Google DeepMind. 2024. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118. ↗
  3. Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. 2023. Grammar-constrained decoding for structured NLP tasks without finetuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10932– 10952, Singapore. Association for Computational Linguistics. find
  4. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2023. explosion/spaCy: v3.7.2: Fixes for APIs and requirements. find
  5. Yan Hu, Qingyu Chen, Jingcheng Du, Xueqing Peng, Vipina Kuttichi Keloth, Xu Zuo, Yujia Zhou, Zehan Li, Xiaoqian Jiang, Zhiyong Lu, Kirk Roberts, and Hua Xu. 2024. Improving large language models for clinical named entity recognition via prompt engineering. Journal of the American Medical Informatics Association, 31(9):1812–1820. ↗
  6. Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. 2024. Evaluating quantized large language models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. ↗
  7. Llama Team, AI @ Meta. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  8. Anita Louis, Alta de Waal, and Cobus Venter. 2006. Named entity recognition in a south african context. In Proceedings of the 2006 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists on IT Research in Developing Countries, SAICSIT ’06, pages 170–179, Somerset West, South Africa. South African Institute for Computer Scientists. find
  9. Microsoft. 2024. Phi-4 technical report. Preprint, arXiv:2412.08905. ↗
  10. Microsoft Research. 2024. Phi-3 technical report: A highly capable language model locally on your phone. Preprint, arXiv:2404.14219. ↗
  11. Saad Obaid ul Islam, Anne Lauscher, and Goran Glavaš. 2025. How much do llms hallucinate across languages? on multilingual estimation of llm hallucination in the wild. Preprint, arXiv:2502.12769. ↗
  12. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108, Online. Association for Computational Linguistics. ↗
  13. Alexander Shan, John Bauer, Riley Carlson, and Christopher Manning. 2023. Do “English” named entity recognizers work well on global englishes? In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11778–11791, Singapore. Association for Computational Linguistics.
  14. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142– 147. ↗
  15. Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023. Gpt-ner: Named entity recognition via large language models. Preprint, arXiv:2304.10428. ↗
  16. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. ↗
  17. Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, Yong Jiang, and Wenjuan Han. 2023. Zero-shot information extraction via chatting with chatgpt. ArXiv, abs/2302.10205. ↗
  18. Ralph Weischedel and 1 others. 2013. OntoNotes release 5.0. Web Download. LDC Catalog No.: LDC2013T19. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ren-2025-slender-structured-outputs-slm-based/