Elamur Research⌕
☰
home / papers / 2026 / IDEAlign: Comparing Ideas of Large Language Models to Domain Experts

IDEAlign: Comparing Ideas of Large Language Models to Domain Experts

Paper · 2026 · HyunJi Nam, Lucía Langlois, Jim Malamut et al.

Summary

LLMs are increasingly used to write free-form, interpretive annotations, such as describing a student's reasoning or suggesting essay revisions. Until now there has been no validated, scalable way to check whether these outputs express the same ideas as expert annotations, rather than merely resembling them in wording, tone or length. The authors treat this idea-level comparison as a task in its own right and propose IDEAlign. In this protocol, experts view three annotations of the same source and pick the one that differs most on an expert-defined criterion. Pooled choices form a probability distribution over the odd item, and automated methods are scored against it with Hellinger distance. They test BLEU, 16 embedding models (with and without PCA post-processing), BERTopic and MALLET-LDA topic divergence, and LLM-as-a-judge on two education datasets: math reasoning in classroom transcripts and inline essay feedback. Most measures agree poorly with experts and are swayed by style and length. LLM-as-a-judge performs best but still trails expert agreement, so the authors recommend it for triage, not as a replacement for human review.

Key points

  • Expert judgments are collected via pick-the-odd-one-out triplets, a relative-judgment design drawing on Thurstone, Bradley-Terry and the "topic intruder" paradigm. Model outputs are compared to human answer distributions using Hellinger distance.
  • Pairwise scores from non-LLM methods are converted into probabilities of each item being chosen as the odd one within a triplet.
  • Directly prompting an LLM to solve the triplet task beats lexical, embedding and topic-based approaches by 11–18% in expert agreement. It works especially well when a clear outlier exists.
  • When used to rank annotator pairs by similarity, LLM-as-a-judge correlates moderately with expert rankings (Spearman ρ 0.58–0.68, p < 0.01) on both datasets.
  • Code and protocol are released at github.com/EduNLP/IDEAlign. The authors advise domain-specific expert validation before applying any of these metrics to new data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as HyunJi Nam, Lucía Langlois, Jim Malamut, Mei Tan and Dorottya Demszky. 2026. IDEAlign: Comparing Ideas of Large Language Models to Domain Experts. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.182

@inproceedings{nam2026idealign,
  title = {{IDEAlign: Comparing Ideas of Large Language Models to Domain Experts}},
  author = {HyunJi Nam, Lucía Langlois, Jim Malamut, Mei Tan and Dorottya Demszky},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.182},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.182}
}

References

67 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anastasios N. Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I. Jordan, and Tijana Zrnic. 2023. Prediction-powered inference. Science, 382(6671):669–674. find
  2. John Annett. 2002. Subjective rating scales: science or art? Ergonomics, 45(14):966–987. PMID: 12569049. find
  3. Anthropic. 2025. System card: Claude opus 4 & claude sonnet 4. find
  4. David Blei, Andrew Ng, and Michael Jordan. 2001. Latent dirichlet allocation. In Advances in Neural Information Processing Systems, volume 14. MIT Press.
  5. Ljubiša Bojić, Olga Zagovora, Asta Zelenkauskaite, Vuk Vuković, Milan Čabarkapa, Selma Veselje-vić Jerković, and Ana Jovančević. 2025. Comparing large language models and human annotators in latent content analysis of sentiment, political leaning, emotional intensity and sarcasm. Scientific r… find
  6. Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324– 345. find
  7. Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-graber, and David Blei. 2009. Reading tea leaves: How humans interpret topic models. In Advances in Neural Information Processing Systems, volume 22. Curran Associates, Inc.
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  9. Alexander R. Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. find
  10. Benjamin Feuer, Micah Goldblum, Teresa Datta, Sanjana Nambiar, Raz Besaleli, Samuel Dooley, Max Cembalest, and John P Dickerson. 2025. Style outweighs substance: Failure modes of LLM judges in alignment benchmarking. In The Thirteenth International Conference on Learning Representations. find
  11. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821.
  12. LearnLM Team Google. 2024. proving gemini for learning. arXiv:2412.16429. find
  13. Learnlm: Im-arXiv preprint Arthur C Graesser, Shulan Lu, George Tanner Jackson, Heather Hite Mitchell, Mathew Ventura, Andrew Olney, and Max M Louwerse. 2004. Autotutor: A tutor with dialogue in natural language. Behavior Research Methods, Instruments, & Computers, 36(2):180–192. find
  14. Ryan Greene, Ted Sanders, Lilian Weng, and Arvind Neelakantan. 2022. New and improved embedding model. https://openai.com/index/ new-and-improved-embedding-model/. ↗
  15. Maarten Grootendorst. 2022. Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794.
  16. Ben Hamner, Jaison Morgan, lynnvandev, Mark Shermis, and Tom Vander Ark. 2012. The hewlett foundation: Automated essay scoring. https://kaggle.com/competitions/asap-aes. Kaggle. ↗
  17. Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2023. Large language models can self-improve. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1051–1068, Singapore. Association for Computational Linguistics.
  18. Mete Ismayilzada, Claire Stevenson, and Lonneke van der Plas. 2025. Evaluating creative short story generation in humans and large language models. Preprint, arXiv:2411.02316. — [arXiv:2411.02316](https://arxiv.org/abs/2411.02316) find
  19. Liwei Jiang, Yuanjun Chai, Margaret Li, Mickel Liu, Raymond Fok, Nouha Dziri, Yulia Tsvetkov, Maarten Sap, Alon Albalak, and Yejin Choi. 2025. Artificial hivemind: The open-ended homogeneity of language models (and beyond). Preprint, arXiv:2510.22954. — [arXiv:2510.22954](https://arxiv.org/abs/2510.22954) find
  20. Ian Jones and Lara Alcock. 2012. Summative peer assessment of undergraduate calculus using adaptive comparative judgement. Mapping university mathematics assessment practices, pages 63–74. find
  21. Sal Khan. 2023. Harnessing gpt-4 so that all students benefit. a nonprofit approach for equal access. Khan Academy Blog. find
  22. Donald Laming. 2003. Human judgment: The eye of the beholder. Cengage Learning EMEA. find
  23. Daniil Larionov, Artem Shelmanov, Elena Chistova, and Ivan Smirnov. 2019. Semantic role labeling with pre-trained language models for known and unknown predicates. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 619–628, Varna, Bul… find
  24. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. Preprint, arXiv:2308.03281.
  25. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  26. Chin-Yew Lin and Franz Josef Och. 2004. ORANGE: a method for evaluating automatic evaluation metrics for machine translation. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 501–507, Geneva, Switzerland. COLING.
  27. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958.
  28. Christopher Mah, Mei Tan, Lena Phalen, Alexa Sparks, and Dora Demszky. 2025. From sentence-corrections to deeper dialogue: Qualitative insights from llm and teacher feedback on student writing. Available at SSRN 5213040. find
  29. Gonzalo Martínez, José Alberto Hernández, Javier Conde, Pedro Reviriego, and Elena Merino-Gómez. 2025. Beware of words: Evaluating the lexical diversity of conversational llms using chatgpt as case study. ACM Transactions on Intelligent Systems and Technology, 16(6):1–15. find
  30. Andrew Kachites McCallum. 2002. A machine learning for language Http://mallet.cs.umass.edu. find
  31. Jiaqi Mu, Suma Bhat, and Pramod Viswanath. 2018. All-but-the-top: Simple and effective post-processing for word representations. Preprint, arXiv:1702.01417. find
  32. Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistic…
  33. Alberto Muñoz-Ortiz, Carlos Gómez-Rodríguez, and David Vilares. 2024. Contrasting linguistic patterns in human and llm-generated news text. Artificial Intelligence Review, 57(10). find
  34. Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madelei… — Text and Code Embeddings by Contrastive Pre-Training
  35. Jun-Ping Ng and Viktoria Abrecht. 2015. Better summarization evaluation with word embeddings for ROUGE. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1925–1930, Lisbon, Portugal. Association for Computational Linguistics.
  36. Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, and Yinfei Yang. 2021. Large dual encoders are generalizable retrievers. Preprint, arXiv:2112.07899.
  37. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… — GPT-4 Technical Report
  38. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train…
  39. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  40. Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Preprint, arXiv:2102.01454.
  41. Alastair Pollitt. 2012. The method of adaptive comparative judgement. Assessment in Education: principles, policy & practice, 19(3):281–300. of the 5th Workshop on Representation Learning for NLP, pages 156–165, Online. Association for Computational Linguistics. find
  42. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. Preprint, arXiv:1908.10084.
  43. Jennifer Routh, Sharmini Julita Paramasivam, Peter Cockcroft, Sarah Wood, John Remnant, Cornélie Westermann, Alison Reid, Patricia Pawson, Sheena Warman, Vishna Devi Nadarajah, et al. 2023. Rating and ranking preparedness characteristics important for veterinary workplace clinical training: a novel… find
  44. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 29971–30004. PMLR.
  45. Kayla Schroeder and Zach Wood-Doughty. 2025. Can you trust llm judgments? reliability of llm-as-a-judge. Preprint, arXiv:2412.12509. — [arXiv:2412.12509](https://arxiv.org/abs/2412.12509) find
  46. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pre-training for language understanding. Advances in neural information processing systems, 33:16857– 16867.
  47. Alexander Spangher, Nanyun Peng, Sebastian Gehrmann, and Mark Dredze. 2024. Do LLMs plan like human writers? comparing journalist coverage of press releases with LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21814–21828, Miami, Florida, USA. A… find
  48. James WA Strachan, Dalila Albergo, Giulia Borghini, Oriana Pansardi, Eugenio Scaliti, Saurabh Gupta, Krati Saxena, Alessandro Rufo, Stefano Panzeri, Guido Manzi, et al. 2024. Testing theory of mind in large language models and humans. Nature Human Behaviour, 8(7):1285–1295. find
  49. Mei Tan, Christopher Mah, and Dorottya Demszky. 2024a. Reframing authority: A computational measure of power-affirming feedback on student writing. In Proceedings of the Eleventh ACM Conference on Learning@ Scale, pages 417–421. find
  50. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  51. Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024b. Large language models for data annotation and synthesis: A survey. Preprint, arXiv:2402.13446.
  52. Vikas Raunak, Vaibhav Kumar, Vivek Gupta, and Florian Metze. 2020. On dimensional linguistic properties of the word embedding space. In Proceedings Pina Tarricone and C Paul Newhouse. 2016. Using comparative judgement and online technologies in the assessment and measurement of creative performance… find
  53. L. Thurstone. 1927. A law of comparative judgment. Psychology Review, 34:273–86. find
  54. Jiayi Zhang, Simon Yu, Derek Chong, Anthony Sicilia, Michael R. Tomz, Christopher D. Manning, and Weiyan Shi. 2025. Verbalized sampling: How to mitigate mode collapse and unlock llm diversity. Preprint, arXiv:2510.01171. find
  55. Somin Wadhwa, Silvio Amir, and Byron C. Wallace. 2024. Revisiting relation extraction in the era of large language models. Preprint, arXiv:2305.05003. find
  56. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. Preprint, arXiv:1904.09675. find
  57. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2024a. Text embeddings by weakly-supervised contrastive pre-training. Preprint, arXiv:2212.03533.
  58. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. Preprint, arXiv:1909.02622.
  59. Rose E. Wang, Qingyang Zhang, Carly Robinson, Susanna Loeb, and Dorottya Demszky. 2024b. Bridging the novice-expert gap via models of decision-making: A case study on remediating math mistakes. Preprint, arXiv:2310.10648. find
  60. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685.
  61. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Preprint, arXiv:2002.10957. find
  62. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. Smarter, better, faster, longer: A modern bidirectional encoder for fast,…
  63. Ruiyang Zhou, Lu Chen, and Kai Yu. 2024. Is LLM a reliable reviewer? a comprehensive evaluation of LLM on automatic paper reviewing tasks. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9340– 9351, T…
  64. Hui Wei, Shenghua He, Tian Xia, Fei Liu, Andy Wong, Jingyang Lin, and Mei Han. 2025. Systematic evaluation of llm-as-a-judge in llm alignment tasks: Explainable metrics and diverse prompt templates. Preprint, arXiv:2408.13006. — [arXiv:2408.13006](https://arxiv.org/abs/2408.13006) find
  65. Qingsong Wen, Jing Liang, Carles Sierra, Rose Luckin, Richard Tong, Zitao Liu, Peng Cui, and Jiliang Tang. 2024. Ai for education (ai4edu): Advancing personalized education with llm and adaptive learning. New York, NY, USA. Association for Computing Machinery. find
  66. Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. 2022. Reframing human-AI collaboration for generating free-text explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie… find
  67. Xinli Yu, Zheng Chen, Yuan Ling, Shujing Dong, Zongyi Liu, and Yanbin Lu. 2023. Temporal data meets llm – explainable financial time series forecasting. Preprint, arXiv:2306.11025. — [arXiv:2306.11025](https://arxiv.org/abs/2306.11025) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nam-2026-idealign-comparing-ideas-large-language/