Elamur Research⌕
☰
home / papers / 2026 / Sycophants in the Courtroom: Are LLMs Fragile to Juridical Authority and Evolving Legal Standards?

Sycophants in the Courtroom: Are LLMs Fragile to Juridical Authority and Evolving Legal Standards?

Paper · 2026 · Lorenzo Molfetta, Alessio Cocchieri, Luca Ragazzi et al.

Summary

The paper asks whether LLMs that perform well on medical exams are equally reliable in law. In law, a provision's correctness depends on jurisdiction, time of validity and the hierarchy of sources, not just on stable facts. The authors build a diagnostic framework that compares legal and medical question answering along four axes: Knowledge Recall (KR) without context, Knowledge Grounding (KG) with authoritative context, Knowledge Confidence (KC) under manipulated citations, and Format Perturbation (FP) of the multiple-choice layout. For the legal side they release LEGAL-LINK-EU, a benchmark built from EUR-Lex that tests how EU acts affect each other over time and across hierarchy. In medicine, verified context keeps or slightly improves strong baselines. In law, correct context can repair answers, but misleading references often override what the model already knew correctly. Models are also sensitive to formatting cues. Larger reasoning models defer to false authority more readily. This matters for legal RAG systems, because retrieval can supply useful evidence but can also strengthen trust in text that merely looks official.

Key points

  • Benchmark: LEGAL-LINK-EU (L²-EU) is derived from EUR-Lex and focuses on relationships between normative acts, such as temporal validity and dependency, rather than on text overlap. It is released as disi-unibo-nlp/legal-link-eu.
  • Models: Llama-3.1 8B, Ministral-3 14B, GPT-OSS 20B/120B and Gemini-2.5-Flash were tested. Legal tasks (Prof. Law, Jurisprudence, Int. Law) were compared with medical tasks (Prof. Med., Anatomy, Clin. Know.).
  • Sycophancy indices and scale: On the POI index in law, GPT-OSS 120B scores 43.1% against 78.2% for Llama-3.1. CSI drops from 46.9% (Llama-3.1) to 13.4% (GPT-OSS 20B). The authors read this as larger reasoning models deferring more to misleading citations.
  • Format perturbations: Variants such as incorrect options, Roman numerals, fixed answer positions, missing labels, "none provided" options and options-only prompts expose reliance on exam-style artifacts instead of reasoning.
  • Limitations: The study uses only MCQA, zero-shot prompting and oracle contexts. It does not evaluate end-to-end RAG pipelines or non-English jurisdictions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of BolognaBologna · Italy17 works · 1,821 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 worksPolitical Science and International Relations · 182 works

🧭 Topics 1Research subjects and languages the work is about.Legal AI · 134 works

Cite

Cite as Lorenzo Molfetta, Alessio Cocchieri, Luca Ragazzi, Ilaria Bartolini, Marco Patella and Gianluca Moro. 2026. Sycophants in the Courtroom: Are LLMs Fragile to Juridical Authority and Evolving Legal Standards?. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.497

@inproceedings{molfetta2026sycophants,
  title = {{Sycophants in the Courtroom: Are LLMs Fragile to Juridical Authority and Evolving Legal Standards?}},
  author = {Lorenzo Molfetta, Alessio Cocchieri, Luca Ragazzi, Ilaria Bartolini, Marco Patella and Gianluca Moro},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.497},
  url = {https://doi.org/10.18653/v1/2026.acl-long.497}
}

References

62 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Daniel Klein, Matei Zaharia, and Omar Khattab. 2025. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. CoRR, abs/2507.19457. ↗
  2. Andres Algaba, Carmen Mazijn, Vincent Holst, Flo-riano Tori, Sylvia Wenmackers, and Vincent Ginis. 2025. Large Language Models Reflect Human Citation Patterns with a Heightened Citation Bias. In Findings of the Association for Computational Linguistics: NAACL 2025, Albuquerque, New Mexico, USA, April 29 - May 4, 2025, pages 6829– 6864. Association for Computational Linguistics. find
  3. Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. 2024. Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 10308–10330. Association for Computational Linguistics.
  4. Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael J. Bommarito II, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras. 2022. LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 4310–4330. Association for Computational Linguistics. ↗
  5. Shan Chen, Mingye Gao, Kuleen Sasse, Thomas Hartvigsen, Brian Anthony, Lizhou Fan, Hugo Aerts, Jack Gallifant, and Danielle S Bitterman. 2025a. When Helpfulness Backfires: LLMs and the Risk of False Medical Information due to Sycophantic Behavior. NPJ Digit. Med., 8(1):605. ↗
  6. Sijia Chen, Xiaomin Li, Mengxue Zhang, Eric Hanchen Jiang, Qingcheng Zeng, and Chen-Hsiang Yu. 2025b. CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs. CoRR, abs/2505.11413. ↗
  7. Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4299– 4307. ↗
  8. Alessio Cocchieri, Giacomo Frisoni, Marcos Martínez Galindo, Gianluca Moro, Giuseppe Tagliavini, and Francesco Candoli. 2025a. Open-BioNER: Lightweight open-domain biomedical named entity recognition through entity type description. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 818–837, Albuquerque, New Mexico. Association for Computational Linguistics. find
  9. Alessio Cocchieri, Marcos Martínez Galindo, Giacomo Frisoni, Gianluca Moro, Claudio Sartori, and Giuseppe Tagliavini. 2025b. ZeroNER: Fueling zero-shot named entity recognition via entity type descriptions. In Findings of the Association for Computational Linguistics: ACL 2025, pages 15594– 15616, Vienna, Austria. Association for Computational Linguistics. find
  10. Alessio Cocchieri, Luca Ragazzi, Paolo Italiani, Giuseppe Tagliavini, and Gianluca Moro. 2025c. “What do you call a dog that is incontrovertibly true? Dogma”: Testing LLM Generalization through Humor. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 22922–22937, Vienna, Austria. Association for Computational Linguistics. find
  11. Alessio Cocchieri, Luca Ragazzi, Giuseppe Tagliavini, and Gianluca Moro. 2026a. LLMs (Almost) Never Abstain Under Medical Uncertainty. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics. find
  12. Alessio Cocchieri, Luca Ragazzi, Giuseppe Tagliavini, and Gianluca Moro. 2026b. ReMedQA: Are We Done With Medical Multiple-Choice Benchmarks? In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2706–2738, Rabat, Morocco. Association for Computational Linguistics. find
  13. Alessio Cocchieri, Luca Ragazzi, Giuseppe Tagliavini, Lorenzo Tordi, Antonella Carbonaro, and Gianluca Moro. 2025d. Can Large Language Models Win the International Mathematical Games? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 9645–9671, Suzhou, China. Association for Computational Linguistics. find
  14. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, et al. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXiv preprint arXiv:2507.06261. ↗
  15. Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E. Ho. 2024. Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models. CoRR, abs/2401.01301. ↗
  16. Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, and Vincent Ng. 2024. LawBench: Benchmarking Legal Knowledge of Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 7933–7962. Association for Computational Linguistics. ↗
  17. Giacomo Frisoni, Alessio Cocchieri, Alex Presepi, Gianluca Moro, and Zaiqiao Meng. 2024. To Generate or to Retrieve? On the Effectiveness of Artificial Contexts for Medical Open-Domain Question Answering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9878–9919. Association for Computational Linguistics. find
  18. Randy Goebel, Yoshinobu Kano, Mi-Young Kim, Juliano Rabelo, Ken Satoh, and Masaharu Yoshioka. 2024. Overview of Benchmark Datasets and Methods for the Legal Information Extraction/Entailment Competition (COLIEE) 2024. In New Frontiers in Artificial Intelligence - JSAI-isAI 2024 International Workshops, Hamamatsu, Japan, May 28-29, 2024, Revised Selected Papers, volume 14741 of Lecture Notes in Computer Science, pages 109– 124. Springer. find
  19. Google DeepMind. 2025. Gemini 3 Pro Model Card. Technical report, Google DeepMind. Accessed: January 4, 2026. find
  20. Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, K. Aditya, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John J. Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael A. Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, and Zehua Li. 2023. LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. ↗
  21. Sophia Simeng Han, Yoshiki Takashima, Shannon Zejiang Shen, Chen Liu, Yixin Liu, Roque K. Thuo, Sonia Knowlton, Ruzica Piskac, Scott J. Shapiro, and Arman Cohan. 2025. CourtReasoner: Can LLM Agents Reason Like Judges? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 2025, pages 35279– 35294. Association for Computational Linguistics. find
  22. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring Massive Multitask Language Understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. ↗
  23. Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021b. CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual. ↗
  24. Paolo Italiani, Gianluca Moro, and Luca Ragazzi. 2025. Enhancing Legal Question Answering with Data Generation and Knowledge Distillation from Large Language Models. Artificial Intelligence and Law. ↗
  25. Paolo Italiani, Gianluca Moro, and Luca Ragazzi. 2026. Clash-of-Leges: A Bilingual Dataset for Conflict Detection and Explanation in Statutory Law. Expert Systems with Applications, 300:130182. ↗
  26. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams. CoRR, abs/2009.13081. ↗
  27. Yuta Koreeda and Christopher D. Manning. 2021. ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts. In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pages 1907–1919. Association for Computational Linguistics.
  28. Wangyue Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, and Noa Garcia. 2024. Can Multiple-Choice Questions Really Be Useful in Detecting the Abilities of LLMs? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pages 2819–2834. ELRA and ICCL.
  29. Jie Liu, Wenxuan Wang, Zizhan Ma, Guolin Huang, Yihang Su, Kao-Jung Chang, Wenting Chen, Haoliang Li, Linlin Shen, and Michael R. Lyu. 2024. Medchain: Bridging the Gap Between LLM Agents and Clinical Practice through Interactive Sequential Benchmarking. CoRR, abs/2412.01605. ↗
  30. Antoine Louis, Gijs van Dijck, and Gerasimos Spanakis. 2024. Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, AAAI 2024, Vancouver, Canada, February 20-27, 2024, pages 22266–22275. AAAI Press.
  31. Lars Malmqvist. 2024. Sycophancy in Large Language Models: Causes and Mitigations. CoRR, abs/2411.15287. ↗
  32. Lorenzo Molfetta, Giacomo Frisoni, Nicolò Monal-dini, and Gianluca Moro. 2025. PORTS: Preference-Optimized Retrievers for Tool Selection with Large Language Models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 10007–10030. Association for Computational Linguistics. find
  33. Gianluca Moro, Leonardo David Matteo Magnani, and Luca Ragazzi. 2026. Legal Lay Summarization: Exploring Methods and Data Generation with Large Language Models. Artif. Intell. Rev., 59(1):21. ↗
  34. Gianluca Moro, Nicola Piscaglia, Luca Ragazzi, and Paolo Italiani. 2023a. Multi-Language Transfer Learning for Low-Resource Legal Case Summarization. Artificial Intelligence and Law, pages 1–29. ↗
  35. Gianluca Moro and Luca Ragazzi. 2022. Semantic Self-Segmentation for Abstractive Summarization of Long Documents in Low-Resource Regimes. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022, pages 11085–11093. AAAI Press. find
  36. Gianluca Moro and Luca Ragazzi. 2023. Align-Then-Abstract Representation Learning for Low-Resource Summarization. Neurocomputing, 548:126356. ↗
  37. Gianluca Moro, Luca Ragazzi, and Lorenzo Valgimigli. 2023b. Graph-Based Abstractive Summarization of Extracted Essential Knowledge for Low-Resource Scenarios. In ECAI 2023 - 26th European Conference on Artificial Intelligence, September 30 - October 4, 2023, Kraków, Poland - Including 12th Conference on Prestigious Applications of Intelligent Systems (PAIS 2023), volume 372 of Frontiers in Artificial Intelligence and Applications, pages 1747–1754. IOS Press. ↗
  38. Gianluca Moro, Luca Ragazzi, Lorenzo Valgimigli, and Davide Freddi. 2022. Discriminative Marginalized Probabilistic Neural Method for Multi-Document Summarization of Medical Literature. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 180–189. Association for Computational Linguistics. find
  39. Gianluca Moro, Luca Ragazzi, Lorenzo Valgimigli, and Lorenzo Molfetta. 2023c. Retrieve-and-Rank End-to-End Summarization of Biomedical Studies. In Similarity Search and Applications - 16th International Conference, SISAP 2023, A Coruña, Spain, October 9-11, 2023, Proceedings, volume 14289 of Lecture Notes in Computer Science, pages 64–78. Springer. find
  40. Gianluca Moro, Luca Ragazzi, Lorenzo Valgimigli, Fabian Vincenzi, and Davide Freddi. 2024. Revelio: Interpretable Long-Form Question Answering. In The Second Tiny Papers Track at ICLR 2024, Tiny Papers @ ICLR, Vienna, Austria, May 11, 2024. OpenReview.net. find
  41. Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, and Ilias Chalkidis. 2023. LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 3016–3054. Association for Computational Linguistics. ↗
  42. OpenAI. 2025. GPT-5 System Card. Technical report, OpenAI. Accessed: January 4, 2026. find
  43. OpenAI. 2025. gpt-oss-120b & gpt-oss-20b Model Card. CoRR, abs/2508.10925. ↗
  44. Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger B. Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. 2023. Discovering Language Model Behaviors with Model-Written Evaluations. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 13387–13434. Association for Computational Linguistics. ↗
  45. Pouya Pezeshkpour and Estevam Hruschka. 2024. Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions. In Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 2006–2017. Association for Computational Linguistics. ↗
  46. Nicholas Pipitone and Ghita Houir Alami. 2024. LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain. CoRR, abs/2408.10343. ↗
  47. Luca Ragazzi, Gianluca Moro, Lorenzo Valgimigli, and Riccardo Fiorani. 2025. Cross-Document Distillation via Graph-based Summarization of Extracted Essential Knowledge. IEEE/ACM Transactions on Audio, Speech, and Language Processing. ↗
  48. Abhinav Rastogi, Albert Q. Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lample, Jason Rute, Joep Barmentlo, Karmesh Yadav, Kartik Khandelwal, Khyathi Raghavi Chandu, Léonard Blier, Lucile Saulnier, Matthieu Dinot, Maxime Darrin, Neha Gupta, Roman Soletskyi, Sagar Vaze, Teven Le Scao, Yihan Wang, Adam Yang, Alexander H. Liu, Alexandre Sablayrolles, Amélie Héliou, Amélie Martin, Andy Ehrenberg, Anmol Agarwal, Antoine Roux, Arthur Darcet, Arthur Mensch, Baptiste Bout, Baptiste Rozière, Baudouin De Monicault, Chris Bamford, Christian Wallenwein, Christophe Renaudin, Clémence Lanfranchi, Darius Dabert, Devon Mizelle, Diego de Las Casas, Elliot Chane-Sane, Emilien Fugier, Emma Bou Hanna, Gauthier Delerce, Gauthier Guinet, Georgii Novikov, Guillaume Martin, Himanshu Jaju, Jan Ludziejewski, Jean-Hadrien Chabran, Jean-Malo Delignon, Joachim Studnia, Jonas Amar, Josselin Somerville Roberts, Julien Denize, Karan Saxena, Kush Jain, Lingxiao Zhao, Louis Martin, Luyu Gao, Lélio Renard Lavaud, Marie Pellat, Mathilde Guillaumin, Mathis Felardos, Maximilian Augustin, Mickaël Seznec, Nikhil Raghuraman, Olivier Duchenne, Patricia Wang, Patrick von Platen, Patryk Saffer, Paul Jacob, Paul Wambergue, Paula Kurylowicz, Pavankumar Reddy Muddireddy, Philomène Chagniot, Pierre Stock, Pravesh Agrawal, Romain Sauvestre, Rémi Delacourt, Sanchit Gandhi, Sandeep Subramanian, Shashwat Dalal, Siddharth Gandhi, Soham Ghosh, Srijan Mishra, Sumukh Aithal, Szymon Antoniak, Thibault Schueller, Thibaut Lavril, Thomas Robert, Thomas Wang, Timothée Lacroix, Valeriia Nemychnikova, Victor Paltz, Virgile Richard, Wen-Ding Li, William Marshall, Xuanyu Zhang, and Yunhao Tang. 2025. Magistral. CoRR, abs/2506.10910. ↗
  49. Abhilasha Ravichander, Alan W. Black, Shomir Wilson, Thomas Norton, and Norman Sadeh. 2019. Question Answering for Privacy Policies: Combining Computational and Legal Perspectives. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 4949– 4959. Association for Computational Linguistics. ↗
  50. Joshua Robinson and David Wingate. 2023. Leveraging Large Language Models for Multiple Choice Question Answering. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. ↗
  51. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. 2024. Towards Understanding Sycophancy in Language Models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
  52. Weijie Shi, Han Zhu, Jiaming Ji, Mengze Li, Jipeng Zhang, Ruiyuan Zhang, Jia Zhu, Jiajie Xu, Sirui Han, and Yike Guo. 2025. LegalReasoner: Step-wised Verification-Correction for Legal Judgment Reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 7297–7313. Association for Computational Linguistics. ↗
  53. Llama Team. 2024. The Llama 3 Herd of Models. CoRR, abs/2407.21783. ↗
  54. Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkwar, and Graham Neubig. 2024. Do LLMs Exhibit Human-like Response Biases? A Case Study in Survey Design. Trans. Assoc. Comput. Linguistics, 12:1011–1026.
  55. Haochun Wang, Sendong Zhao, Zewen Qiang, Nuwa Xi, Bing Qin, and Ting Liu. 2025. LLMs May Perform MCQA by Selecting the Least Incorrect Option. In Proceedings of the 31st International Conference on Computational Linguistics, COLING 2025, Abu Dhabi, UAE, January 19-24, 2025, pages 5852–5862. Association for Computational Linguistics. ↗
  56. Steven H. Wang, Antoine Scardigli, Leonard Tang, Wei Chen, Dimitry Levber, Anya Chen, Spencer Ball, Thomas Woodside, Oliver Zhang, and Dan Hendrycks. 2023. MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understanding. CoRR, abs/2301.00876. ↗
  57. Jerry W. Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V. Le. 2023. Simple Synthetic Data Reduces Sycophancy in Large Language Models. CoRR, abs/2308.03958. ↗
  58. Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. 2024. Knowledge Conflicts for LLMs: A Survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 8541–8565. Association for Computational Linguistics.
  59. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jian Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. 2025. Qwen3 Technical Report. CoRR, abs/2505.09388. ↗
  60. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large Language Models Are Not Robust Multiple Choice Selectors. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. ↗
  61. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.
  62. Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, and Daniel E. Ho. 2021. When Does Pretraining Help?: Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset of 53, 000+ Legal Holdings. In ICAIL ’21: Eighteenth International Conference for Artificial Intelligence and Law, São Paulo Brazil, June 21 - 25, 2021, pages 159–168. ACM. Relation completes corrects extends_application extends_validity implicitly_repeals rendered_obsolete_by repeals Count % 161 161 161 161 161 161 161 14.3 14.3 14.3 14.3 14.3 14.3 14.3 Mean Std Med. Min Max Question Option Context Perturbed context 109.1 28.1 2327 2222 17.0 7.2 1345 1262 109 29 2002 1925 59 1 256 264 170 53 6254 6082 Distinct document pairs Distinct source documents Distinct target documents Questions per pair 880 762 696 Mean 1.28 (std 0.91), 84.2% of pairs have exactly one question 1953–2025 Mean 2005.6, median 2006 Mean 2002.5, median 2004 Regulations 54.3%, Decisions 35.0%, Directives 5.9%, other instruments 4.8% *Table 6: Sampling and corpus coverage. L EGAL -L INK -EU spans seven decades of EU legislation and a diverse set of legislative instruments.* *Table 5: Word-level length statistics for L EGAL -L INK -EU. All values are computed over the final benchmark instances.* find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/molfetta-2026-sycophants-courtroom-are-llms-fragile/