Elamur Research⌕
☰
home / papers / 2026 / When LLMs Disagree with Human Experts: Understanding LLM Annotation Failures in Nutrition Misinformation through Hierarchical Error Analysis using Seed Oil Narratives

When LLMs Disagree with Human Experts: Understanding LLM Annotation Failures in Nutrition Misinformation through Hierarchical Error Analysis using Seed Oil Narratives

Paper · 2026 · Vishwaa Shah, Indika Kahanda and Andrea Arikawa

Summary

The paper examines how reliable open LLMs are as annotators of health misinformation in a specialized field, where scientific evidence is nuanced. The authors gathered 169 English Instagram captions about seed oils from high-reach accounts. Three trained nutrition experts independently labeled each caption as Credible or Misinformation, using the 2020–2025 U.S. Dietary Guidelines as the reference. A majority vote produced the gold labels. Five instruction-tuned open-weight models then received knowledge-augmented Chain-of-Thought prompts and returned reasoning, a binary label and a confidence score. Besides standard classification metrics, the authors coded every model–expert disagreement with a three-level hierarchical error taxonomy covering:

  • the direction of the error,
  • its mechanism,
  • contributing factors such as hedged claims, evidence cues and context.

The models showed recurring weaknesses. They often misread nuanced or partly accurate claims, and they frequently gave high confidence to wrong labels. The study contributes a reproducible workflow and an openly released dataset that show where LLM labels diverge from expert judgment. This can guide human-in-the-loop annotation design for health and nutrition content.

Key points

  • Dataset: 169 Instagram captions on seed oils, collected May 28 – June 4, 2025, from public accounts with more than 5,000 followers. Collection used fresh accounts and a VPN to limit personalization. Roughly 71% of posts fall in the positive class.
  • Expert labels: Three nutrition experts annotated each caption, reaching Fleiss' κ = 0.813. The final label was decided by majority vote.
  • Models evaluated: Qwen3-4B, Mistral-7B, Llama-2-7B, Zephyr-7B and GPT-OSS-13B, all under one knowledge-augmented CoT prompting setup. Annotating the full set took about 30 to 90 minutes per model.
  • Error taxonomy: Three hierarchical levels (error direction, error mechanism, contributing factors). It captures pragmatic and linguistic sources of mismatch, such as claim hedging, rather than factuality alone.
  • Release and limitations: Data, annotations and error analysis are available on Zenodo under CC BY 4.0. The study is limited to text only, English only and U.S.-guideline grounding, uses binary labels, and has a small sample.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of North FloridaJacksonville · United States2 works

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Vishwaa Shah, Indika Kahanda and Andrea Arikawa. 2026. When LLMs Disagree with Human Experts: Understanding LLM Annotation Failures in Nutrition Misinformation through Hierarchical Error Analysis using Seed Oil Narratives. Proceedings of the 20th Linguistic Annotation Workshop (LAW XX).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.law-main.6

@inproceedings{shah2026when,
  title = {{When LLMs Disagree with Human Experts: Understanding LLM Annotation Failures in Nutrition Misinformation through Hierarchical Error Analysis using Seed Oil Narratives}},
  author = {Vishwaa Shah, Indika Kahanda and Andrea Arikawa},
  booktitle = {Proceedings of the 20th Linguistic Annotation Workshop (LAW XX)},
  year = {2026},
  doi = {10.18653/v1/2026.law-main.6},
  url = {https://doi.org/10.18653/v1/2026.law-main.6}
}

References

11 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sergio Segado-Fernández, Beatriz Jiménez-Gómez, Pedro Jesús Jiménez-Hidalg, María del Carmen Lozano-Estevan, and Iván Herrera-Peco. 2025. Disinformation about diet and nutrition on social networks: a review of the literature. Nutricion hospitalaria, 42(2). find
  2. Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Mohammadmasiha Zahedivafa, Juan D Bermeo, Maria Korobeynikova, and Fabrizio Gilardi. 2025. Open-source llms for text annotation: a practical guide for model setting and fine-tuning. Journal of Computational Social Science, 8(1):17. find
  3. Wen-Ying Sylvia Chou, Anna Gaysynsky, and Joseph N Cappella. 2020. Where we go from here: health misinformation on social media. find
  4. Joseph Balagtas and Elijah Bryant. 2025. Consumer food insights. Technical report, Center for Food Demand Analysis and Sustainability, Purdue University. find
  5. Dongmei Tan, Yi Huang, Ming Liu, Ziyu Li, Xiaoqian Wu, and Cheng Huang. 2025. Identification of online health information using large pretrained language models: Mixed methods study. Journal of Medical Internet Research, 27:e70733. find
  6. Brian Dean. 2025. Instagram demographic statistics: How many people use instagram in 2024? Accessed: 2025-04-25. find
  7. Dietary Guidelines for Americans. 2020. Di-Zhen Tan, Dawei Li, Song Wang, Alimohammad etary guidelines for americans, 2020-Beigi, Bohan Jiang, Amrita Bhattacharjee, Man-2025 and online materials. https: sooreh Karami, Jundong Li, Lu Cheng, and Huan //www.dietaryguidelines.gov/resources/ Liu. 2024. L… find
  8. Mitra, and Zhengjie Miao. 2024. Human-llm collaborative annotation through effective verification of llm labels. In Proceedings of the 2024 CHI conference on human factors in computing systems, pages 1–21. find
  9. Akshay Goel, Almog Gueta, Omry Gilon, Chang Liu, Sofia Erell, Lan Huong Nguyen, Xiaohong Hao, Bolous Jaber, Shashir Reddy, Rupesh Kartha, and 1 others. 2023. Llms accelerate annotation for medical information extraction. In machine learning for health (ML4H), pages 82–100. PMLR. find
  10. Andy Wai Kan Yeung, Anela Tosevska, Elisabeth Klager, Fabian Eibensteiner, Christos Tsagkaris, Emil D Parvanov, Faisal A Nawaz, Sabine Völkl-Kernstock, Eva Schaden, Maria Kletecka-Pulker, and 1 others. 2022. Medical and health-related misinformation on social media: bibliometric study of the scienti… find
  11. Prajwol Lamichhane, Indika Kahanda, Andrea Arikawa, Charlotte Martin, Maribel Garcia, Camila Figueiredo, and Haivan Benjamin. 2025. Exploring the feasibility of identifying nutrition misinformation on social media. In Proceedings of the ACM/IEEE International find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shah-2026-when-llms-disagree-human-experts/