Elamur Research⌕
☰
home / papers / 2026 / Treating Decoder-Only LLMs as Encoders: A Simple and Effective Fine-tuning Approach for Named Entity Recognition

Treating Decoder-Only LLMs as Encoders: A Simple and Effective Fine-tuning Approach for Named Entity Recognition

Paper · 2026 · Ken Yano and Hiroya Takamura

Summary

Decoder-only LLMs are trained with causal masking, so each token's representation sees only what comes before it. Token-level tasks like named entity recognition need context on both sides, which is why encoders such as BERT usually do better. Yano and Takamura make decoder LLMs work as encoders. They swap the causal mask for full bidirectional attention in every decoder block, replace the language-modeling head with a token classification head, and train it on BIO labels.

They test seven models from 0.5B to 8B parameters, drawn from Gemma, Qwen2.5, Llama3.2 and Llama3.1. Each is trained only on the BC5CDR (biomedical) and CoNLL2003 (general) training sets, with both full fine-tuning and LoRA. Bidirectional attention alone does not beat strong encoders. With a well-chosen LoRA setup, the best models outperform BERT, RoBERTa and DeBERTa, though they stay below published SOTA.

This matters because existing LLMs can become drop-in encoders for sequence labeling without auxiliary data or new architectures. The trade-off is substantially higher training and inference cost.

Key points

  • Models are labeled LS-FA-LLM (full attention) or LS-CA-LLM (causal attention). Baselines are bert-base-cased, roberta-base and deberta-v3-large, plus BioBERT_v1.1 for BC5CDR. Published SOTA from OpenMED (BC5CDR) and ACE (CoNLL2003) is cited for reference.
  • LoRA is preferred over full fine-tuning because full fine-tuning tends to overfit these small datasets, especially with large models.
  • The LoRA sweep shows a clear gain from targeting all linear modules rather than only q,k,v,o:
    • LS-FA-Llama3.1-8B reaches up to 0.904 micro F1 on BC5CDR and 0.932 on CoNLL2003.
    • LS-FA-Llama3.2-3B reaches up to 0.900 and 0.935.
  • In error analysis, LS-FA-Llama3.1-8B makes fewer errors than DeBERTa in every SPAN, TYPE and DETECTION subtype. Its causal-attention counterpart performs poorly, which points to the causal mask as the bottleneck.
  • On a single H200, the largest LS-FA models (Gemma-7B, Llama3.1-8B) take roughly 10× BERT's inference time.
  • The approach cannot handle nested or discontinuous entities.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬National Institute of Advanced Industrial Science and TechnologyTsukuba · Japan18 works · 2,760 citations🔬Artificial Intelligence Research CenterTokyo · Japan9 works · 1,564 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksNamed entity recognition · 214 works

Cite

Cite as Ken Yano and Hiroya Takamura. 2026. Treating Decoder-Only LLMs as Encoders: A Simple and Effective Fine-tuning Approach for Named Entity Recognition. BioNLP 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bionlp-1.25

@inproceedings{yano2026treating,
  title = {{Treating Decoder-Only LLMs as Encoders: A Simple and Effective Fine-tuning Approach for Named Entity Recognition}},
  author = {Ken Yano and Hiroya Takamura},
  booktitle = {BioNLP 2026},
  year = {2026},
  doi = {10.18653/v1/2026.bionlp-1.25},
  url = {https://doi.org/10.18653/v1/2026.bionlp-1.25}
}

References

4 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arthur Amalvy, Vincent Labatut, and Richard Dufour. 2023. The role of global and local context in named entity recognition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 714–722, Toronto, Canada. Association for Computation… find
  2. Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. Llm2vec: Large language models are secretly powerful text encoders. Preprint, arXiv:2404.05961.
  3. Yubo Chen, Chuhan Wu, Tao Qi, Zhigang Yuan, and Yongfeng Huang. 2020. Named entity recognition BC5CDR Model Baseline Encoders BERT RoBERTa DeBERTa IFT-LLM GPT-4.1 IFT-Gemma-2B IFT-Gemma-7B IFT-Qwen2.5-0.5B IFT-Qwen2.5-7B IFT-Llama3.2-1B IFT-Llama3.2-3B IFT-Llama3.1-8B LS-FA-LLM LS-FA-Gemma-2B LS-FA-… find
  4. Precision (set) Recall (set) F1 (set) Precision (set) Recall (set) F1 (set) Full Full Full 0.861 0.882 0.887 0.877 0.896 0.896 0.869 0.889 0.892 0.914 0.917 0.915 0.914 0.921 0.916 0.914 0.919 0.915 Zero-shot LoRA LoRA LoRA LoRA LoRA LoRA LoRA 0.670 0.870 0.878 0.803 0.881 0.866 0.879 0.901 0.505 0.… find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yano-2026-treating-decoder-only-llms-as/