Elamur Research⌕
☰
home / topics / Language models & pretraining / page 7

🧠 Language models & pretraining

1,079 papers · page 7 of 12 · 526–625 · the most cited first · all topics

Models that learn a language from raw text, and how they are pretrained, tuned and prompted.

1 … 6 7 8 … 12

  1. LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic ProversTheo X. Olausson, Alex Gu, Benjamin Lipkin, et al.2023 · 171 citations
  2. OpenChat: Advancing Open-source Language Models with Mixed-Quality DataGuan Wang, Sijie Cheng, Xianyuan Zhan, et al.2023 · 171 citations
  3. KernelBench: Can LLMs Write Efficient GPU Kernels?Anne Ouyang, Simon Guo, Simran Arora, et al.2025 · 170 citations
  4. Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMsSimone Balloccu, Patrícia Schmidtová, Mateusz Lango, et al.2024 · 170 citations
  5. LESS: Selecting Influential Data for Targeted Instruction TuningMengzhou Xia, Sadhika Malladi, Suchin Gururangan, et al.2024 · 170 citations
  6. AudioPaLM: A Large Language Model That Can Speak and ListenPaul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, et al.2023 · 169 citations
  7. Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-SpecificityCunxiang Wang, Xiaoze Liu, Yuanhao Yue, et al.2023 · 169 citations
  8. Low-Resource Languages Jailbreak GPT-4Zheng-Xin Yong, Cristina Menghini, Stephen H. Bach2023 · 168 citations
  9. ShortGPT: Layers in Large Language Models are More Redundant Than You ExpectXin Men, Mingyu Xu, Qingyu Zhang, et al.2024 · 167 citations
  10. The Capacity for Moral Self-Correction in Large Language ModelsDeep Ganguli, Amanda Askell, Nicholas Schiefer, et al.2023 · 167 citations
  11. Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4Kent K. Chang, Mackenzie Cramer, Sandeep Soni, et al.2023 · 166 citations
  12. Deduplicating Training Data Mitigates Privacy Risks in Language ModelsNikhil Kandpal, Eric Wallace, Colin Raffel2022 · 166 citations
  13. Structure-Grounded Pretraining for Text-to-SQLXiang Deng, Ahmed Hassan Awadallah, Christopher Meek, et al.2021 · 166 citations
  14. Generative Representational Instruction TuningNiklas Muennighoff, Hongjin Su, Liang Wang, et al.2024 · 165 citations
  15. Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge ConflictsJian Xie, Kai Zhang, Jiangjie Chen, et al.2023 · 165 citations
  16. Large Language Model UnlearningYuanshun Yao, Xiaojun Xu, Yang Liu2023 · 165 citations
  17. Binding Language Models in Symbolic LanguagesZhoujun Cheng, Tianbao Xie, Peng Shi, et al.2022 · 165 citations
  18. “I'm sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor DatasetEric Michael Smith, Melissa Hall, Melanie Kambadur, et al.2022 · 165 citations
  19. On Generative Spoken Language Modeling from Raw AudioKushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, et al.2021 · 165 citations
  20. Sustainable Modular Debiasing of Language ModelsAnne Lauscher, Tobias Lueken, Goran Glavaš2021 · 165 citations
  21. Do All Languages Cost the Same? Tokenization in the Era of Commercial Language ModelsOrevaoghene Ahia, Sachin Kumar, Hila Gonen, et al.2023 · 164 citations
  22. From System 1 to System 2: A Survey of Reasoning Large Language ModelsZhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, et al.2025 · 163 citations
  23. Improve Mathematical Reasoning in Language Models by Automated Process SupervisionLiangchen Luo, Yinxiao Liu, Rosanne Liu, et al.2024 · 163 citations
  24. AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource LanguagesAbteen Ebrahimi, Manuel Mager, Arturo Oncevay, et al.2022 · 163 citations
  25. Long-form factuality in large language modelsJerry Wei, Chengrun Yang, Xinying Song, et al.2024 · 162 citations
  26. Catastrophic Jailbreak of Open-source LLMs via Exploiting GenerationYangsibo Huang, Samyak Gupta, Mengzhou Xia, et al.2023 · 162 citations
  27. GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak PromptsJiahao Yu, Xingwei Lin, Zheng Yu, et al.2023 · 162 citations
  28. LooGLE: Can Long-Context Language Models Understand Long Contexts?Jiaqi Li, Mengmeng Wang, Zilong Zheng, et al.2023 · 162 citations
  29. Impact of Pretraining Term Frequencies on Few-Shot Numerical ReasoningYasaman Razeghi, Robert L Logan IV, Matt Gardner, et al.2022 · 162 citations
  30. Probing What Different NLP Tasks Teach Machines about Function Word ComprehensionNajoung Kim, Roma Patel, Adam Poliak, et al.2019 · 161 citations
  31. CultureLLM: Incorporating Cultural Differences into Large Language ModelsCheng Li, Mengzhou Chen, Jindong Wang, et al.2024 · 160 citations
  32. Orca 2: Teaching Small Language Models How to ReasonArindam Mitra, Luciano Del Corro, Shweti Mahajan, et al.2023 · 160 citations
  33. The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False DatasetsSamuel Marks, Max Tegmark2023 · 160 citations
  34. Reasoning with Language Model Prompting: A SurveyShuofei Qiao, Yixin Ou, Ningyu Zhang, et al.2022 · 160 citations
  35. Larger-Scale Transformers for Multilingual Masked Language ModelingNaman Goyal, Jingfei Du, Myle Ott, et al.2021 · 160 citations
  36. Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language ModelsBofei Gao, Feifan Song, Zhe Yang, et al.2024 · 159 citations
  37. When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMsRyo Kamoi, Yusen Zhang, Nan Zhang, et al.2024 · 159 citations
  38. L-Eval: Instituting Standardized Evaluation for Long Context Language ModelsChenxin An, Shansan Gong, Ming Zhong, et al.2023 · 159 citations
  39. PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel ComputationWei Zeng, Xiaozhe Ren, Teng Su, et al.2021 · 159 citations
  40. GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via CipherYouliang Yuan, Wenxiang Jiao, Wenxuan Wang, et al.2023 · 158 citations
  41. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt CompressionHuiqiang Jiang, Qianhui Wu, Xufang Luo, et al.2023 · 158 citations
  42. GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation ModelsAohan Zeng, Xin Lv, Qinkai Zheng, et al.2025 · 157 citations
  43. From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction TuningMing Li, Yong Zhang, Zhitao Li, et al.2023 · 157 citations
  44. SmoothLLM: Defending Large Language Models Against Jailbreaking AttacksAlexander Robey, Eric Wong, Hamed Hassani, et al.2023 · 157 citations
  45. The Linear Representation Hypothesis and the Geometry of Large Language ModelsKiho Park, Yo Joong Choe, Victor Veitch2023 · 157 citations
  46. Unsupervised Text Style Transfer using Language Models as DiscriminatorsZichao Yang, Zhiting Hu, Chris Dyer, et al.2018 · 157 citations
  47. DataComp-LM: In search of the next generation of training sets for language modelsJeffrey Li, Alex Fang, Georgios Smyrnis, et al.2024 · 156 citations
  48. Canine: Pre-training an Efficient Tokenization-Free Encoder for Language RepresentationJonathan H. Clark, Dan Garrette, Iulia Turc, et al.2022 · 156 citations
  49. LayerSkip: Enabling Early Exit Inference and Self-Speculative DecodingMostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, et al.2024 · 155 citations
  50. WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMsSeungju Han, Kavel Rao, Allyson Ettinger, et al.2024 · 155 citations
  51. Probing Pre-Trained Language Models for Cross-Cultural Differences in ValuesArnav Arora, Lucie-aimée Kaffee, Isabelle Augenstein2023 · 155 citations
  52. Moshi: a speech-text foundation model for real-time dialogueAlexandre Defossez, Laurent Mazare, Manu Orsini, et al.2024 · 154 citations
  53. ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMsJustin Chen, Swarnadeep Saha, Mohit Bansal2024 · 154 citations
  54. Large Language Models with Controllable Working MemoryDaliang Li, Ankit Singh Rawat, Manzil Zaheer, et al.2023 · 154 citations
  55. Leveraging Large Language Models for Multiple Choice Question AnsweringJoshua Robinson, Christopher Michael Rytting, David Wingate2022 · 154 citations
  56. Stop Overthinking: A Survey on Efficient Reasoning for Large Language ModelsYang Sui, Yu-Neng Chuang, Guanchu Wang, et al.2025 · 153 citations
  57. Can GPT-3 Perform Statutory Reasoning?Andrew Blair-Stanek, Nils Holzenberger, Benjamin Van Durme2023 · 153 citations
  58. Label Words are Anchors: An Information Flow Perspective for Understanding In-Context LearningLean Wang, Lei Li, Damai Dai, et al.2023 · 153 citations
  59. Graph Pre-training for AMR Parsing and GenerationXuefeng Bai, Yulong Chen, Yue Zhang2022 · 153 citations
  60. SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingTianzhe Chu, Yuexiang Zhai, Jihan Yang, et al.2025 · 151 citations
  61. Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language ModelsTianyi Tang, Wenyang Luo, Haoyang Huang, et al.2024 · 151 citations
  62. Training Language Models to Self-Correct via Reinforcement LearningAviral Kumar, Vincent Zhuang, Rishabh Agarwal, et al.2024 · 151 citations
  63. The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language ModelsJunyi Li, Jie Chen, Ruiyang Ren, et al.2024 · 150 citations
  64. AnnoLLM: Making Large Language Models to Be Better Crowdsourced AnnotatorsXingwei He, Zhenghao Lin, Yeyun Gong, et al.2023 · 150 citations
  65. Pre-Training BERT on Arabic Tweets: Practical ConsiderationsAhmed Abdelali, Sabit Hassan, Hamdy Mubarak, et al.2021 · 150 citations
  66. Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language ModelsWenhao Yu, Hongming Zhang, Xiaoman Pan, et al.2024 · 149 citations
  67. Language Model Tokenizers Introduce Unfairness Between LanguagesAleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, et al.2023 · 149 citations
  68. Investigating Pretrained Language Models for Graph-to-Text GenerationLeonardo F. R. Ribeiro, Martin Schmitt, Hinrich Schütze, et al.2021 · 149 citations
  69. SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language ModelsXiaoxuan Wang, Ziniu Hu, Pan Lu, et al.2023 · 148 citations
  70. BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and LanguagesJunho Myung, Nayeon Lee, Yi Zhou, et al.2024 · 147 citations
  71. Measuring short-form factuality in large language modelsJason Wei, Nguyen Karina, Hyung Won Chung, et al.2024 · 147 citations
  72. rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep ThinkingXinyu Guan, Li Lyna Zhang, Yifei Liu, et al.2025 · 146 citations
  73. Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative DecodingJun Zhang, Jue Wang, Huan Li, et al.2024 · 146 citations
  74. Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAsMicrosoft, Abdelrahman Abouelenin, Atabak Ashfaq, et al.2025 · 145 citations
  75. How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data CompositionGuanting Dong, Hongyi Yuan, Keming Lu, et al.2024 · 145 citations
  76. Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative DecodingHeming Xia, Zhe Yang, Qingxiu Dong, et al.2024 · 145 citations
  77. Context-faithful Prompting for Large Language ModelsWenxuan Zhou, Sheng Zhang, Hoifung Poon, et al.2023 · 145 citations
  78. RA-DIT: Retrieval-Augmented Dual Instruction TuningXi Victoria Lin, Xilun Chen, Mingda Chen, et al.2023 · 145 citations
  79. SyntaxGym: An Online Platform for Targeted Evaluation of Language ModelsJon Gauthier, Jennifer Hu, Ethan Wilcox, et al.2020 · 145 citations
  80. Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with NothingZhangchen Xu, Fengqing Jiang, Luyao Niu, et al.2024 · 144 citations
  81. Unified Demonstration Retriever for In-Context LearningXiaonan Li, Kai Lv, Hang Yan, et al.2023 · 144 citations
  82. In-Context Learning for Few-Shot Dialogue State TrackingYushi Hu, Chia-Hsuan Lee, Tianbao Xie, et al.2022 · 144 citations
  83. Is GPT-3 a Good Data Annotator?Bosheng Ding, Chengwei Qin, Linlin Liu, et al.2022 · 144 citations
  84. MiniLLM: On-Policy Distillation of Large Language ModelsYuxian Gu, Li Dong, Furu Wei, et al.2023 · 143 citations
  85. Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM ReasoningShenzhi Wang, Le Yu, Chang Gao, et al.2025 · 142 citations
  86. Justice or Prejudice? Quantifying Biases in LLM-as-a-JudgeJiayi Ye, Yanbo Wang, Yue Huang, et al.2024 · 142 citations
  87. In-Context Learning Creates Task VectorsRoee Hendel, Mor Geva, Amir Globerson2023 · 142 citations
  88. PMC-LLaMA: Towards Building Open-source Language Models for MedicineChaoyi Wu, Weixiong Lin, Xiaoman Zhang, et al.2023 · 142 citations
  89. ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool EmbeddingsShibo Hao, Tianyang Liu, Zhen Wang, et al.2023 · 142 citations
  90. TALM: Tool Augmented Language ModelsAaron Parisi, Yao Zhao, Noah Fiedel2022 · 142 citations
  91. Rethinking embedding coupling in pre-trained language modelsHyung Won Chung, Thibault Févry, Henry Tsai, et al.2020 · 142 citations
  92. JudgeLM: Fine-tuned Large Language Models are Scalable JudgesLianghui Zhu, Xinggang Wang, Xinlong Wang2023 · 141 citations
  93. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelMichael Hanna, Ollie Liu, Alexandre Variengien2023 · 140 citations
  94. A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence GenerationNeeraj Varshney; Wenlin Yao; Hongming Zhang; Jianshu Chen;, Dong Yu2023 · 139 citations
  95. The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context LearningBill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, et al.2023 · 139 citations
  96. LongRoPE: Extending LLM Context Window Beyond 2 Million TokensYiran Ding, Li Lyna Zhang, Chengruidong Zhang, et al.2024 · 138 citations
  97. DeepInception: Hypnotize Large Language Model to Be JailbreakerXuan Li, Zhanke Zhou, Jianing Zhu, et al.2023 · 138 citations
  98. Exploring Length Generalization in Large Language ModelsCem Anil, Yuhuai Wu, Anders Andreassen, et al.2022 · 138 citations
  99. R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement LearningHuatong Song, Jinhao Jiang, Yingqian Min, et al.2025 · 137 citations
  100. Investigating Data Contamination in Modern Benchmarks for Large Language ModelsChunyuan Deng, Yilun Zhao, Xiangru Tang, et al.2024 · 137 citations

1 … 6 7 8 … 12

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/topics/language-models/page/7/