Elamur Research⌕
☰
home / papers / 2026 / Lost in Formatting: How Output Formats Skew LLM Performance on Information Extraction

Lost in Formatting: How Output Formats Skew LLM Performance on Information Extraction

Paper · 2026 · Rishi Ravikumar, Nuhu Ibrahim and Riza Batista-Navarro

Summary

LLM-based information extraction systems are usually fine-tuned to produce one output format, often picked by habit. The authors test whether this choice matters. Their framework covers three sequence-labelling tasks, each with two benchmarks:

  • Event Detection: ACE2005 and PHEE
  • Named Entity Recognition: OntoNotes-5.0 and BC5CDR
  • Semantic Role Labelling: SpRL-2012 and NounAtlas

They define six information-equivalent output formats: two standoff variants (one in JSON, one as tuples), two inline variants (one using XML tags, one linear), plus a Column format and a Freetext format. Eight decoder-only instruction models (3B–32B parameters) were fine-tuned across these settings, in more than 280 experiments.

Changing only the format can move F1 by over 40% for the same model. No format is best everywhere. The winner depends on dataset traits such as span density, span length and document length, and on the model family. Token-level formats lead by a small margin on average, and larger models are less sensitive to format.

The authors conclude that output format should be tuned empirically like a hyperparameter. They provide practical guidelines for choosing one.

Key points

  • Models evaluated: Qwen2.5 (3B, 14B, 32B), Phi-3.5-mini, Mistral-7B-v0.3, Llama3.1-8B, Mistral-Nemo and OLMo-2-0325-32B. Code is published on GitHub (rishi-ravikumar/OutputFormattingExperiments).
  • Effect size: on BC5CDR, a format switch alone caused the model to miss over 40% of entity mentions.
  • Format trade-offs:
    • Token-level formats (Inline Linear, Column) have the highest mean F1 and tend to give higher recall. They suit dense annotations and long texts.
    • Standoff and Freetext formats suit sparse annotations and long spans, where they make fewer boundary errors.
  • Robustness: measured as the F1 gap between the best and worst format, it improves with model size. Model family also matters: Mistral 7B is unusually stable for its size.
  • Efficiency and prior work:
    • Standoff JSON and Tuple outputs are much more compact, so they are cheaper to train and run.
    • The results contradict earlier claims: the token-level format favoured by Raman et al. is not always best, and the top format reported by Guo et al. ranks among the weakest here.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksInformation extraction · 167 works

Cite

Cite as Rishi Ravikumar, Nuhu Ibrahim and Riza Batista-Navarro. 2026. Lost in Formatting: How Output Formats Skew LLM Performance on Information Extraction. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.256

@inproceedings{ravikumar2026lost,
  title = {{Lost in Formatting: How Output Formats Skew LLM Performance on Information Extraction}},
  author = {Rishi Ravikumar, Nuhu Ibrahim and Riza Batista-Navarro},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.256},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.256}
}

References

51 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hammaad Adam, Junjing Lin, Jianchang Lin, Hillary Keenan, Ashia Wilson, and Marzyeh Ghassemi. 2024. Clinical information extraction with large language models: A case study on organ procurement. AMIA Annual Symposium Proceedings, 2024:115–123. find
  2. Ben Athiwaratkun, Cicero Nogueira dos Santos, Jason Krone, and Bing Xiang. 2020. Augmented natural language for generative sequence labeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 375–385, Online. Association for Computational Lingui… find
  3. John Dagdelen, Alexander Dunn, Sanghoon Lee, Nicholas Walker, Andrew S. Rosen, Gerbrand Ceder, Kristin A. Persson, and Anubhav Jain. 2024. Structured information extraction from scientific text with large language models. Nature Communications, 15(1):1418.
  4. Angel Daza and Anette Frank. 2018. A sequence-to-sequence model for semantic role labeling. In Proceedings of the Third Workshop on Representation Learning for NLP, pages 207–216, Melbourne, Australia. Association for Computational Linguistics. find
  5. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: efficient finetuning of quantized llms. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  7. George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The automatic content extraction (ACE) program – tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC‘04), Lisbo…
  8. David Dukić and Jan Snajder. 2024. Looking right is sometimes right: Investigating the capabilities of decoder-only LLMs for sequence labeling. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14168–14181, Bangkok, Thailand. Association for Computational Linguistics. find
  9. Daniel Gildea and Daniel Jurafsky. 2000. Automatic labeling of semantic roles. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 512–520, Hong Kong. Association for Computational Linguistics.
  10. Ralph Grishman and Beth Sundheim. 1996. Message Understanding Conference- 6: A brief history. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics.
  11. Aynur Guluzade, Naguib Heiba, Zeyd Boukhers, Florim Hamiti, Jahid Hasan Polash, Yehya Mohamad, and Carlos A. Velasco. 2025. ELMTEX: Fine-Tuning LLMs for Structured Clinical Information Extraction. A Case Study on Clinical Reports, page 181–185. Springer Nature Switzerland. find
  12. Biyang Guo, He Wang, Wenyilin Xiao, Hong Chen, ZhuXin Lee, Songqiao Han, and Hailiang Huang. 2024. Sample design engineering: An empirical study on designing better fine-tuning samples for information extraction with LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Languag… find
  13. Han He and Jinho D. Choi. 2023. Unleashing the true potential of sequence-to-sequence models for sequence tagging and structure parsing. Transactions of the Association for Computational Linguistics, 11:582–599. find
  14. Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X Wang, and Sadid Hasan. 2024. Does prompt formatting have any impact on llm performance? Preprint, arXiv:2411.10541. — [arXiv:2411.10541](https://arxiv.org/abs/2411.10541) find
  15. Kuan-Hao Huang, I-Hung Hsu, Tanmay Parekh, Zhiyu Xie, Zixuan Zhang, Prem Natarajan, Kai-Wei Chang, Nanyun Peng, and Heng Ji. 2024. TextEE: Benchmark, reevaluation, reflections, and future challenges in event extraction. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12… find
  16. Nancy Ide, Christian Chiarcos, Manfred Stede, and Steve Cassidy. 2017. Designing Annotation Schemes: From Model to Representation, pages 73–111. find
  17. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  18. Soo joon Choi and Ji jun Park. 2025. Harnessing generative llms for enhanced financial event entity extraction performance. Preprint, arXiv:2504.14633. — [arXiv:2504.14633](https://arxiv.org/abs/2504.14633) find
  19. Martin Josifoski, Nicola De Cao, Maxime Peyrard, Fabio Petroni, and Robert West. 2022. GenIE: Generative information extraction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4626–4643, Seattle… find
  20. Parisa Kordjamshidi, Steven Bethard, and Marie-Francine Moens. 2012. SemEval-2012 task 3: Spatial role labeling. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth… find
  21. LDC. 2005. Ace (automatic content extraction) english annotation guidelines for events. Accessed: 2025-02-12. find
  22. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240.
  23. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the…
  24. Jiangnan Li, Yice Zhang, Bin Liang, Kam-Fai Wong, and Ruifeng Xu. 2023. Set learning for generative information extraction. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13043–13052, Singapore. Association for Computational Linguistics. find
  25. Jiao Li, Yueping Sun, Robert J. Johnson, Dan Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan P. Davis, Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu. 2016. Biocreative v cdr task corpus: a resource for chemical disease relation extraction. Database. find
  26. Yuanye Liu, Jiahang Xu, Li Lyna Zhang, Qi Chen, Xuan Feng, Yang Chen, Zhongxin Guo, Yuqing Yang, and Peng Cheng. 2025. Beyond prompt content: Enhancing llm performance via content-format integrated prompt optimization. Preprint, arXiv:2502.04295. — [arXiv:2502.04295](https://arxiv.org/abs/2502.04295) find
  27. Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi Chen. 2021. Text2Event: Controllable sequence-to-structure generation for end-to-end event extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the…
  28. Yi-Long Lu, Chunhui Zhang, and Wei Wang. 2025. Systematic bias in large language models: Discrepant response patterns in binary vs. continuous judgment tasks. Preprint, arXiv:2504.19445. — [arXiv:2504.19445](https://arxiv.org/abs/2504.19445) find
  29. Meta. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  30. Microsoft. 2024. Phi-3 technical report: A highly capable language model locally on your phone. Preprint, arXiv:2404.14219.
  31. George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. 1993. A semantic concordance. In Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21-24, 1993.
  32. Mistral AI. 2024. Mistral nemo. Accessed: 2025-05-05. find
  33. Roberto Navigli, Marco Lo Pinto, Pasquale Silvestri, Dennis Rotondi, Simone Ciciliano, and Alessandro Scirè. 2024. NounAtlas: Filling the gap in nominal semantic role labeling. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1… find
  34. NuMind. 2024. Nuextract: A foundation model for structured extraction. numind.ai/blog/nuextract-a-foundation-model-for-structured-extraction. Accessed: 2025-05-10. find
  35. Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cícero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation between augmented natural languages. CoRR, abs/2101.05779. find
  36. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng… find
  37. Karthik Raman, Iftekhar Naim, Jiecao Chen, Kazuma Hashimoto, Kiran Yalasangi, and Krishna Srinivasan. 2022. Transforming sequence tagging into a Seq2Seq task. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11856–11874, Abu Dhabi, United Arab Emirates… find
  38. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. Preprint, arXiv:2310.11324. find
  39. Zhaoyue Sun, Jiazheng Li, Gabriele Pergola, Byron Wallace, Bino John, Nigel Greene, Joseph Kim, and Yulan He. 2022. PHEE: A dataset for pharmacovigilance event extraction from text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5571–5587, Abu Dhabi,… find
  40. Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. 2024. Let me speak freely? a study on the impact of format restrictions on large language model performance. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Tr… find
  41. Shubo Tian, Qiao Jin, Lana Yeganova, Po-Ting Lai, Qingqing Zhu, Xiuying Chen, Yifan Yang, Qingyu Chen, Won Kim, Donald C Comeau, Rezarta Islamaj, Aadit Kapoor, Xin Gao, and Zhiyong Lu. 2024. Opportunities and challenges for chatgpt and large language models in biomedicine and health. Briefings in Bi… find
  42. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142– 147.
  43. Asahi Ushio and Jose Camacho-Collados. 2021. T-NER: An all-round python library for transformer-based named entity recognition. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 53–62, Online. Association for… find
  44. Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong, Jie Tang, and Dawn Song. 2022. DeepStruct: Pre-training of language models for structure prediction. In Findings of the Association for Computational Linguistics: ACL 2022, pages 803–823, Dublin, Ireland. Association for Computational Linguistics. find
  45. Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, Guoyin Wang, and Chen Guo. 2025. GPT-NER: Named entity recognition via large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4257–4275, Albuquerque, New Mexico. Assoc…
  46. Xiao Wang, Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, Jihua Kang, Jingsheng Yang, Siyuan Li, and Chunsai Du. 2023. Instructuie: Multi-task instruction tuning for unified information extraction. Preprint, arXiv:2304.08085.
  47. Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. 2013. Ontonotes release 5.0. LDC2013T19, Linguistic Data Consortium. find
  48. Tongtong Wu, Fatemeh Shiri, Jingqi Kang, Guilin Qi, Gholamreza Haffari, and Yuan-Fang Li. 2023. KC-GEE: knowledge-based conditioning for generative event extraction. World Wide Web, 26(6):3983–3999. find
  49. Xinglin Xiao, Yijie Wang, Nan Xu, Yuqi Wang, Hanxuan Yang, Minzheng Wang, Yin Luo, Lei Wang, Wenji Mao, and Daniel Zeng. 2024. Yayi-uie: A chat-enhanced instruction tuning framework for universal information extraction. Preprint, arXiv:2312.15548. — [arXiv:2312.15548](https://arxiv.org/abs/2312.15548) find
  50. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…
  51. Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. 2021. A unified generative framework for various NER subtasks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ravikumar-2026-lost-formatting-how-output-formats/